Recruitering

I know, first we hang all the lawyers. But soon after that, maybe when the telephone sanitizers have stopped kicking (a little mission creep never hurt anyone, right?) we line up the recruiters and introduce them to a little high school physics. F=MA and G=Mm over R-squared something something and you can’t push a rope. Oh, and when you meet the Big Guy, you should know that we made a couple of minor changes to your resume, so just go along with the modifications, okay? It’ll be totally cool, don’t worry, nobody ever checks that stuff.

But . . . I know, you’re a /good/ recruiter, right? You never cold-emailed an employer and said “I’ve got a bunch of super red hot engineers” and then started pimping knuckle-draggers incapable of spelling ‘C’ without looking up the answer on The Google. No, you’re not one of /those/ recruiters, and that makes me happy because you are special and sure, we’ll work with you. Let’s see some of your best clients.

And suddenly it’s like that sequence in Mrs Doubtfire where Robin Williams is faking calls from various inappropriate prospective nannies in order to nail the nanny job himself, except that the people on the other end of the phone are a lot less funny than Robin Williams.

“Print the numbers from 1 to 100? Sure? First we fill a table with numbers 1 to 100, then we run SELECT on the table. Easy!”

“This is like code, right? It’s been a while. Let’s see. Um. FOR. No, IF. Wait, I know this. PRINT 1 TO 100. Okay, that was easy. Next question?”

“Here in Havenukeistan we have many fine count. You relocate, yes? Then I write count for you, best you ever see.”

—-

This works in the other direction, too. 30 years ago a recruiter sent me to the hinterlands outside of Minneapolis, and in that flat desolation surrounded by wheat fields was a small brick building with a bunch of workers in it. I walked in, introduced myself as the candidate, and spent an hour laughing with someone while we tried to figure out what my recruiter figured a game programmer and systems guy would do at a company that made sewing machines and that didn’t even have a computer on the premises. (A couple of later start-ups I was at made me wonder if bolting together light machinery and sleeping in a cow barn would have been a good choice of career change).

Ever wonder how those little bobbin things work? Fascinating, now /there/ is genius we should all aspire to.

Posted in Uncategorized | 6 Comments

A bird on the wall …

It was mid-afternoon and someone was poking me. “Hey, are you awake?” they whispered. I blinked and raised my chin.

“Huh?”

“Ralph wants to know what database we’ve chosen. What database did we choose?”

“Grughnk. Guhbbah.” I’d nearly fallen asleep in the marathon marketing meeting again. “Um, we decided on, um, Squorical.”

“What?”

“Orasqueakle Server. Yeah, that was it.”

“Oh God.”

Many eyes were on me. Many of the eyes had suits and ties attached to them, and they wore expectant looks. I put down the notebook I’d been doodling satanic cartoons in and lurched from my chair. “Sorry, I’ve got to –” I said, and waved generally toward the door. (“Oraskwackle? Really?”)

Ralph the Marketing Guy said, “Great idea. Ten minute bio-break everyone!” He was chipper and ready for another hour at least. I wanted to throw him off our balcony.

We’d been in that meeting room for nearly two hours, discussing nonsense and useless stuff (along the lines of Douglas Adams: “Okay, if you’re so smart, YOU tell us what color the database tables should be!”) and I needed to be doing designing and coding rather than listening to things that made my shit itch. It was the third multi-hour marathon meeting with Marketing that week, and I was nearly at the end of my rope. One of the of the topics under discussion was why engineering was behind schedule.

—-

Our little group of programmers often walked to lunch. I’m not sure where we went the next day; it might have been to the borderline not-very-good chinese place down the hill that had wonderful mis-spellings on its menu (my favorite dish was Prok Noddles), or it might have been the Insanely Hot chinese place that was in an old IHOP A-frame building (they had a great spicy soup that would spray a perfect Gaussian distribution of hot pepper oil on the paper tablecloth as you ate from the bowl). All I remember for sure is that on the return walk we passed the shops on the street as we always did, but for some reason I paused just outside the clock shop, which I’d never really paid attention to before.

A spark of evil flared in my head. “You guys go on, I’m going in here.” I got a couple of strange looks, but nobody followed me in. I spent about a hundred dollars.

—-

The next morning the marathon meeting started on time, but there was a new attendee. I had nailed it to the wall. It said “Cuckoo!” ten times.

“What is that?” one of the marketeers asked. [This question pretty much defines most marketing for me, by the way].

“Just a reminder,” I said with slight smile and a tone that implied it was no big deal. (I’d talked with our CEO about it earlier and he had said okay, but I think I was going to do it anyway).

The rest of the usual attendees filtered in and the meeting started. The clock said “Bong!” at fifteen minutes past the hour, which got a couple of raised eyebrows, and it bonged again on the half hour and a quarter-to. At 10:59 it made a “brrrr-zup!” sound, as if it was getting ready for some physical effort, then at 11:00 the bird popped out, interrupting a PowerPoint presentation that I’ll never be able to recall, and went “CUCKOO! CUCKOO!” about a million times. Well, only eleven, but it certainly felt like a long, long time. The guy who had been talking had trouble remembering what he’d been saying. Everyone else had been expecting the display, but the speaker, one of our serial meeting-stretchers, had been lost in his own blather.

The meeting broke early that day.

The next day, too.

The hours we spent in that room no longer passed anonymously. The clock’s smaller noises, as it prepared to say “Bong”, and its more dramatic preparations for each hour’s Big Show were now part of the agenda. We knew time was passing. The meetings did become shorter. We had an interesting time explaining the clock to customers (for this was a small company with only a couple of meeting rooms), but in general people from outside understood, and often laughed in approval.

The clock was a relatively cheap model that had to be wound every day. I usually got in to work pretty early and wound it, but occasionally I forgot to, or worked remotely and didn’t get to the office at all. Somehow the clock never ran down. It turned out that our office manager was winding it, and occasionally our CEO would wind it, too.

I knew I’d gotten the message across when one of the marketing guys looked up at the bird (which had just announced 3PM) and said, “I hate that thing.” And I smiled, and the meeting ended soon after, and I got up and left the room and went to my keyboard and wrote some more code.

[I left that start-up a few months later. I still have the clock.]

Posted in Uncategorized | 7 Comments

Keeping sharp

A quick story (and a true one, I promise).

The Team is at lunch. Say, ten or twelve software engineers sitting at a high table in the Commons at Microsoft, eating sandwiches and chips and soup and so forth. Oh, and there’s a new guy, fresh out of college.

“The thing is,” says Phil to the FNG, “You really have to be prepared in meetings. They might ambush you and want a schedule, or ask when that really hard bug is going to be fixed, or get you to commit to working on some scrum metrics or something, and you have to be ready.”

And with that, he takes out his rather large Kershaw knife, flicks the blade open and bangs it down on the table.

And without hesitation, about six of us take our own knives out, flip them open, and bang them on the table, too. They’re pointy.

“You have to be ready.”

I’ve not seen someone’s eyes that large at lunch, before or since.

Posted in Uncategorized | 5 Comments

The eternal war against the guys in building B

I wish that I could travel back in time, hand a copy of Horowitz’s book The Art of Electronics to younger me and say, “Read this thing. It will make talking to hardware engineers a bunch easier.”

I realize now that all the time the H/W engineers were wringing their hands and pulling their beards and basically being insufferable about how many watts something was consuming, what they were really saying was, “This stupid thing is drawing two fucking amps at 5 volts and we are going bugfuck crazy supplying that over a two meter wire and into a case which you say can’t have a fan. Can you save us from thermal and emotional meltdown by running about 50Mhz slower?” And we software engineers didn’t get it, and our reply was a haughty “No, we’re a platform, we might need those CPU cycles someday,” and we didn’t know that we’d come this close to being knifed in the kidneys after getting off an elevator reprogrammed to go to the darkened sub-sub basement of the Lair of the Hardware Engineers (building B), from which no software engineer has ever escaped, topologically intact.

So the next time I hear a switching power supply whine in protest, I will think of it as the squeals of pain of the engineers whose life I turned into a living hell because of my lack of appreciation for P = IV. I’m truly sorry. I wasn’t thinking. (And this is just the first chapter of that book).

Then again, we can do a crapload of really cool things with another ten percent of CPU reservation…

->ding<-

Posted in Uncategorized | 1 Comment

Patching the Newton

This one ends with me going into a Good Guys electronics store in a suburb of Boston, flashing my employee badge and saying “I’m from Apple. Take me to your Newtons.” I wanted to use a deep G-man voice, but I didn’t have the build for it. Also, I was wearing the slightly redonkulous khaki-pants and polo shirt that we’d been instructed to wear for the Newton debut the next day, and I looked like a rah-rah Apple sales guy.  Appearances are deceiving.

It’s August, 1993, the day before Apple’s Newton MessagePad goes on sale, and overnight we’ve heard that some people who managed to get units ahead of time are having trouble with them, and someone heard that someone said they thought they had a unit whose software wasn’t the right version. There’s a little uncertainty. Since we shipped the final Newton ROM in June we’ve spent all summer fixing bugs and making patches.  The patches are applied in the factory, toward the end of the production line, and they fix critical bugs. If the units don’t have these fixes, the Newt won’t work very well.

The patches live in the battery protected low-power RAM of the Newton, and they’re theoretically immortal as long as power holds out. This is why the battery compartment has a wacky mechanical locking system meant to discourage people from simultaneously removing both the main and the backup batteries. It’s a byzantine contraption of sliders and buttons molded in Holy Shit Yellow, and it’s meant to scare people into being cautious. But early adopters aren’t necessarily normal people, and maybe they’ve been mucking with stuff they shouldn’t be. Who knows?

As the morning progresses it becomes clear that some number of the Newtons that are already at stores and about to be sold either didn’t get patched at the factory, or (worse) are losing their patches in transit. So here I am, cruising around Boston in a crappy rental car, fumbling with maps and getting lost on streets that are not laid out in a grid. It’s also about 90 degrees and humid, reminding me of why I don’t live on the east coast any more.

I find the Good Guys store manager. “I’m from Apple, and you have a bunch of Newtons that you can’t sell unless we take a look at them first.”

“Um, okay.”

They lead me into into the back room, and there’s a cage with about twenty Newtons in it. It’s the first time I’ve seen one in an actual retail box. It’s kind of neat. We made this? Holy crap, people are about to buy them.

“I’m going to have to open each one and inspect it, okay?”

“Sure.” No pushback at all. They get a stockroom kid to help me. We open each box, I check the version number that each newt displays on its startup screen, and (lo!) there are a couple whose versions are “1.0”. I record the serial numbers of these units, stick the upgrader PCMCIA card into them (hacked up Very Quickly by Newtonites who didn’t get to go to Boston), and verify that the units take the patch. We re-box everything (I resist the urge to include a scribbled note, something like “Inspected by #18706″, my Apple badge number).  Half an hour later, we’re done.

“That’s it. You can sell these now.”

The Guy from Apple exits Good Guys, checks it off the list, and proceeds to the next store on his assignment. G-Man is surprised what we was able to do by flashing his Apple badge and adopting some bluster; just walk in, make some demands and start cracking open merchandise. That’s kind of disturbing, really.

There are perhaps a dozen of us spread out over the Boston metro area, patching Newtons that have somehow lost their minds. We don’t know the cause of the failure yet. We report serial numbers back to Cupertino.  It’s the usual mode of operation of Newton; shit happens, we fix it just in time, and there are unsung heros. A number of engineers back in Infinite Loop did an all-nighter last night and made that magic re-patching PCMCIA card. And here I am stuck in Boston traffic. It’s hawt.

None of this is going to matter.

—-

Back in February or March, the Newton OS group knew we had to solve a tough problem. The Newton’s ROM actually is read-only. The days of shipping flashable devices as a common practice were a few years off.  The Newton’s OS is decided forever by its silicon mask when the chip is fabbed, and it can’t be changed, ever. Also, the lead time for ROMs is on the order of ten weeks, so the software will have to be done earlier, which means it will have more bugs. Huh.

So, how do you fix bugs in a ROM, if you can’t change the image? The basic idea is that you litter the code with indirect jumps that go through a jump table that’s been copied to RAM. When you need to patch something, chances are that you can do tricky and unnatural things to get control at the right spot, fix things up and continue.

The brain-dead way to do this is to give each and every function in the ROM (except for the critical startup stuff) a jump table entry.  You copy the whole table to RAM when the system starts up. For the first Newton the size of the jump table was about 40K, which is about twice the budget we had for patches (the first Newton had 512K of memory, a fair amount of which was reserved for storage of user data, with the rest used for NewtonScript heaps, screen buffers, thread stacks, networking, working storage for handwriting recognition and so on).

Okay, no problem; the Newton has an MMU, and we used it extensively already. So we put the jump table into ROM, and then only RAM-back the pages of it that contain function stubs that we need to modify. But this doesn’t work because pretty often a function you need to patch will be in a page you haven’t used yet. Even a moderate number of patches will quickly drag in the whole jump table again, and you haven’t even included the actual patch code yet.

And that’s where we were roughly in March, knowing that we had a problem, but with nobody really working on it because all we had were the old, dumb ideas that didn’t work. Time hadn’t run out yet, but people outside the team were getting a little antsy.

—-

The OS group was in one “pod” of Infinite loop’s building One (we were one of the first groups to move in, having done our bit of purgatory in an off-campus building on Bubb road for a number of years). The floor layout was a big improvement over a sea of cubicals; the building architects had designed very workable areas in each corner of the building.  Each corner pod had about a dozen offices (with doors!) surrounding a living-room sized area that had couches and *lots* of whiteboard space. If you left your office door open you could halfway participate in the conversations happening in the central area, and if you wanted privacy you just closed your door (which was a nice, solid hunk of wood that stopped sound pretty well). The setup was designed to let people both collaborate and to get quiet think-time. After years of working in cubicals, it was great.

The discussion one afternoon was yet another bull session on how to do patches. I was typing away (bringing our flash object store to life) and not listening too closely, but . . . to this day I don’t know exactly where the idea came from. We were big on randomized policy in Newton because Bob, one of our OS heavies, maintained that it was the ultimate in fairness. Maybe that was it.

I popped my head out of my office and let my mouth talk by itself. “Why don’t we randomize the jump table?”

“What?”

It wasn’t clear in my head yet. This half-baked crazy idea was zooming around upstairs, trying to get out. I went to the whiteboard and started drawing boxes. Whenever an idea is trying to escape, start by drawing boxes and arrows — it’s great bait. The little critters can’t resist a whiteboard with unfulfilled boxes and arrows.

“We randomize the function placement in virtual memory.” Boxes. “We spread them out over a few megabytes but tile them sort of skewed so they use only 40K or so of actual physical memory.” Arrows. “So when you patch a function, you copy it to RAM and map it where it needs to be, but the rest of the page is available in virtual space for the code.”

One of my cow-orkers caught the idea and ran with it. He wasn’t a whiteboard kind of guy, though, so someone else picked up a pen and drew what he was saying.

Greg: “Yeah, we can re-use the same physical page over and over again until we run out of space.” Boxes. “We write a tool to figure out the optimal packing.”

Bob: “We need to make sure that patches are really well protected.  REALLY well. Maybe we shouldn’t ever map them anywhere as writable.”

Daniel: “We need some kind of patch-building tool. And we have to read it from somewhere to get patches installed, and mapped-in when we boot. I’ll do the patch manager work.”

[conversations not accurately remembered, but that was the gist]

In short order we figured out the work, and a couple of weeks later all the tooling and build stuff in place. When we told some of the old Macintosh patching veterans about the technique the reaction was usually a thoughtful look, then a little giggling, and then they wanted to try to write a patch.

Usually a patch would first be written by the engineer who fixed the problem in the mainline sources, and it would go to one of the patching gurus (hi, Andy!) who could distill someone else’s hard-won 16 instruction patch into three instructions of such utter genius and evil that shivers would run down your spine.

I was busy on other stuff and only had time to watch the work happen, but it was pretty cool.

—-

[Here's what was trying to come out. In virtual space, patches would be in a big honking address space, with functions distributed around it (not really randomly, but that was how we got the idea). Each line of dashes below is a page boundary:

 jump A
 -
 -
 -
 ----
 -
 jump B
 -
 -
 ----
 -
 -
 jump C
 -
 ----
 -
 -
 -
 jump D

Note that the virtual address of each jump table entry crawls forward a little on each successive page. So physically we can pack the page like this:

 jump A
 jump B
 jump C
 jump D

and map it multiple times.

Now let's say that we need to patch functions B and C. We'd make a new page, distributed as the patch itself, that contains the new code for B and C at the right locations:

 fix-for-B
 jump to fix-for-B  (was 'jump B')
 inline-fix-for-C   (was 'jump C')
 more-fix-for-C

and map this twice, once for B and once for C, leaving mappings for A and D unchanged and still using the ROM pages:

jump A
 -
 -
 -
 ----
 fix-for-B
 jump to fix-for-B
 (ignored) inline-fix-for-C
 (ignored) more-fix-for-C
 ----
 (ignored) fix-for-B
 (ignored) jump to fix-for-B
 inline-fix-for-C
 more-fix-for-C
 ----
 -
 -
 -
 jump D

The actual fixes might involve jumping to the original code, or could be *significantly* more tricky and convoluted. What mattered was that the patch pages could be packed with code for fixes for different routines. Also, the skewed jump table format was a little more complex, so that we could often include the fixes inline in just a few instructions.

Anyway, the technique saved us a significant amount of RAM, and let us get ROMs out in time for the Big Day.]

—-

Ultimately the Newton was a market failure. If you look at the competition at the time we did maybe five things wrong that the other guys got right or chose not to try to address:

1. Price point. The Newton was about a thousand dollars. Ouch. The Palm Pilot was about $300.

2. Form factor. The Newton was large and weighed about a pound. The other guy fit in a shirt pocket.

3. Handwriting recognition wasn’t quite there (and the production Newton had a digitizer noise issue that we found out about only later that Fall, which dramatically reduced the recognition rate). The Pilot used a much more primitive letter-by-letter system, but people got it and it worked well after a little training. The 2.0 version of the Newton did much better here (but I had left Apple before this shipped).

4. No real apps . . . and you had to pay Apple a commission. For a totally new platform, charging your developers to develop software is an uphill battle. (Of course, all this changed with the iPhone, but the demand was very high). The commission idea was just another stupid and reality-disconnected idea from upper management; instead of attracting developers (who already had to pony up over a thousand bucks for a dev kit) I think it discouraged folks from even considering developing for the Newton.

5. The “MessagePad” wasn’t all that good at actually messaging anything. It was awkward to connect a Newton to anything (via a cable or PCMCIA card), and the modes of communication were either inconvenient or expensive (wireless pay-by-the-kilobyte). The Newton was about 15 years ahead of the technology curve here.

6. Releasing on a magic date. The Newton probably needed another three months or so of bake time before we shipped it. By November or so we’d patched nearly all of the software issues that critics (rightly) complained about, and I like to think that the reception would have been much kinder if we’d worked just a little longer. To this day I have a deep mistrust of magic ship dates and the motivation behind them.

7. Language? I actually think that NewtonScript was a great idea, but the tools were rocky and never really approached the quality that Microsoft soon showed off in Visual BASIC. Apple never released a native dev kit for the Newton, but given the state of its native runtime (gnarly) it’s probably a good idea. In any event, developing on the Pilot was a lot like programming a 68000-based Mac, and pretty familiar to many programmers, while NewtonScript was something from another world (even if it did have many cool aspects).

I don’t regret working on the Newton at all. The people I worked with were top-notch, I learned a hell of a lot about technology and business, and I’m still pretty proud of all the neat stuff we put in under the hood. Ultimately I don’t think that tweaking what we got wrong would have saved the product, but it might have left more of a mark than it did.

Some day I’ll talk about Wayfarer… :-)

—-

I should add: I don’t think we ever found out why those initial units lost their patches. I think we figured they never had them to begin with (it *was* a new production line of a 1.0 product, after all).

Newtonites, feel free to correct any of this.

Posted in Uncategorized | 8 Comments

Things that suck

Java.

Java updates that utterly break already teetering remote access utilities. The fact that these utilities were written in Java at all is mind boggling. I also reserve a special place in hell for the firmware engineers who thought they could write web pages and Java applets and then put them into flash that, if you fat-finger an update — in the vain hope that something or other got fixed — or experience a network blip at the wrong time, toasts your remote access to the machine, whereupon you get to drive to the datacenter and spend quality time with USB thumb drives. I had to make a DOS boot thumb drive today. What the hell?

IDEs written in Java.

Windows software that doesn’t uninstall, and that leaves behind little bits and pieces that throw up ads and nagware until you hunt them down and kill them with the registry editor. (None of this stuff is written in Java, as far as I know, but I am an inclusive kind of guy).

Drivers who nearly run you over in a crosswalk while talking on a cell phone. It looked like an Android phone, so there you go again: Java. It’s trying to fucking kill me and I hates it.

Posted in Uncategorized | 5 Comments

LFE (Looking For Editor)

When you say your editor has Emacs bindings, unless you’ve correctly implemented ^U don’t even bother to make the claim. You don’t get it, and you probably never will, and if you argue with me about it I will simply walk away, shaking my head. Also, if your editor automatically makes a blank line at the end of a file, I will find you and teach you the errors of your ways (hint: patching binaries with Emacs is not uncommon).

That is all.

[Epsilon, my favorite editor, hasn't had an update since 2006 or so, and it's getting a little long in the tooth. Also, it's not totally wonderful on MacOS. It's still perfectly serviceable, but I'd like to have some of the creature comforts of Sublime Text, or Komodo but none of them are really quite there. Surprisingly, XCode is the best one I've found, but it doesn't grok ^U and probably can't be made to, and this makes my fingers itch.]

Posted in Uncategorized | 5 Comments

When rocks were young

Many “old iron” stories here. Back in the day, *real* computers blew up, shot out sparks and spewed smoke, and could be fixed with a good whack in the right spot. Not like today, when you can shut down entire cities from the comfort of your living room with a simple firmware security hole.

Meta-nostalgia is when you find one of your own nostalgiac postings in someone else’s nostalgia.

Posted in Uncategorized | 1 Comment

Haskell is useless

A fun little video of Simon Peyton Jones on useful -vs- useless languages, and trends in language design.

 

Posted in Uncategorized | 1 Comment

Missed opportunity

A little while ago, I finally got The Call.

“Hello?” I heard a gabble of voices on the other end, obviously a boiler room call.

“Sir? Mumble garble findows bhmflg wugga woop.”

“Excuse me? I can hardly hear you with all the noise on your end.”

“Sir? This is Miriam, calling from Microsoft Windows Support. How are you today?”

Oh my.

“I’m doing great! How are you?”

She hung up immediately.

It must have been something in my voice. I must have sounded too eager. I should have said something like, “I’m doing lousy, my computer isn’t working and I need expert to help me install spyware on my Windows PC,” or would that have been leading too much?

I don’t know why they even bother calling this area code.

Posted in Uncategorized | 1 Comment