Dadhacker

It used to be pretty easy to tell when you were done with a piece of embedded software: You wrote the bits to mag tape, the kind on a reel, boxed it up with a big check and mailed it off to chip company in California (if you were already in Cali you just went down the street a few blocks and saved some postage).  Then you waited a couple of months, and if the mask ROMs you got back were in runnable condition, you sold it.

These days, with everything connected and usually updatable, there’s no real “We’re done” event. Oh, there’s a ship party, but usually we go back to fixing bugs after that. If your toaster is suddenly making charcoal briquettes because of a bug in its internal clock, you can just hook the thing up to the internet and grab the latest firmware. The team that did the toaster software is on the hook for fixes.

Briinngggg!!!! [1]

“Urm . . . three bloody o’clock in the — wazzup?”

“We have eighteen hundred bug reports of the iToast 5G sending fountains of flames to the ceiling in the past few hours. Can you come in and fix it?”

“Er, why do I need to drive in? I can log in and fix it from here.”

“Also, explosions. We really want everyone away from those toasters.  You should probably avoid the kitchen and just hit your house’s main breaker on your way here.”

I’m not necessarily saying that the quality of things has dropped — it’s freaking awesome being able to add functionality to products in the field, or being able to fix crap that you fat-fingered at 2AM – but somehow, I miss being done.

—-

[1] Yes, I know that telephones don’t go “Brrrinnngggg” any more.

Dear Blizzard,

If you design a popular game that needs to be permanently connected in order to play, please start with an architecture that can handle enough user load at launch time.

Signed,

Someone who spent sixty bucks on some currently unusable bits

I’ve been riding motorcycles for over 20 years. How I started went like this.

“Want to go to lunch?” I asked Munch, one of my cow-orkers. This fellow doesn’t have a car and rides exclusively, every day of the year (which is possible in Silly Valley).  He’s one of the best street riders I know.

“Okay,” He said. He handed me his spare helmet. “Let’s go.”

“Uh,” I had to think about it for second, “Okay.” I’d never been on the back of a motorcycle before.

We collected Jeff, another of the every-day riders in our group, and went to the parking lot. I’d seen their bikes out there, but hadn’t really appreciated them. Now I was going to get a much closer look.  Munch was riding a BMW K100 RT, a touring model, and Jeff had a smaller K75.

Munch supervised me as I put on the helmet. It fit okay. I didn’t know how to secure the D-rings, so he demonstrated. He then did some stuff at the handlebars which started the bike (I was surprised at how easy that seemed), got on and rocked the bike off its center stand. He turned to me with some instructions.  “You,” he said sternly, “Are like a sack of potatoes. You don’t help me and you don’t *not* help me, either. We’re going to lean, and you can’t fight it. Got it?”

I nodded. Nodding with a helmet on felt weird. I got on the passenger seat and found a place to hang on. I didn’t want to *hug* my cow-orker, so I found handholds behind me.

We headed to a burrito joint a couple miles away. I thought we’d take side streets out of pity for a newbie, but Munch got right on the highway. I hung into the bike, and every time he accelerated I thought I would fall off backwards (there was no danger of this, it just /felt/ like it).

The freeway exit, a nice sweeper, was exhilarating.

We ate lunch (“Burritos as big as your head” at a place where the staff barely spoke english) and talked about bikes.  I knew very little about them and I had lots of questions. When we finished, we visited California BMW just down the street, where I wandered, lost and curious, between the unfamiliar bikes while Munch and Jeff talked to the people running the shop.  Their business finished, we zoomed back to work. I returned the helmet to Munch.

When I got back to my cubical, I called the Motorcycle Safety Foundation and enrolled in a beginner’s training course. A couple months later I spent two weekends and two evenings learning how to ride. It was great fun.

It had been suggested in the MSF course that our first bike should be something we’d be happy with for a while, rather than something “small and safe” (since small doesn’t necessarily equate to safe), so I decided to get a medium-size bike. About a week after finishing the course, I had a bike, a Honda Transalp, 600cc.

After a few more weeks of practice riding (some with Munch, a lot more in the hills around Silly Valley), I took the DMV motorcycle test.  It was a cinch compared to the MSF training. When I finished, the examiner who had run me through the obstacles asked if I’d taken MSF.  I said so, and the examiner told me that the MSFers had a far easier time passing the test.

—-

Since then I’ve been through five or six bikes (mostly BMWs), and logged a few hundred thousand miles on two wheels, mostly in the hills and up and down the California/Oregon/Washington coast. For a few years I didn’t even own a car, I biked everywhere.

—-

Some other memories:

Jeff bought a new Ducati from a dealer in Livermore. Munch and I rode him over, and we took the bikes back over Mines road, up the back of Mount Hamilton and back down into San Jose; a nice mix of twisties and fast sweepers. During this ride, pieces of Jeff’s new bike kept falling off: A glued-on reflector here, some other widget there. By the time we reached the First Friendly Gas Station in San Jose, Jeff’s bike had lost the use of its kickstand, and he had to fuel while straddling the bike to keep it upright. Munch and I bid him farewell on his long ride up to his home in San Francisco; we didn’t follow him to pick up any more stray parts.

- – - -

One weekend we took a ride to Monterey, and returned on 101. Jeff was upset at all the freeway miles, so we turned west into the hills south of San Jose.  Jeff was on a new BMW K100RS, Munch was on his K100RT, and I had my Transalp. Pretty soon we started hitting dirt roads; I was in my element, but the K-bikes were having a hard time. A dog or two chased us. Finally we reached the top of a ridge and found paved road, where we parked and just doubled over in laughter. K bikes are not dirt bikes, but with a sufficiently good rider, they _can_ be.

- – - -

Munch was one of the best programmers I know. I worked with him for several years at Apple, and I when I moved groups I was always trying to get him to come to the group I was in (and later, to one or two of the startups I was at).

Once, he needed to make some tables for a 68000 disassembler. He decided to learn Prolog, and in a day or two he’d generated the code for the disassembler using a Prolog interpreter he’d found. Good programmers write tools. Great programmers learn new stuff at the same time.

- – - -

When I joined Apple in 1987, engineering samples of the new 21″ monochrome monitors were just arriving in various groups. Our group got one, and Munch got it. These displays were an unbelievably crisp white, and the amount of screen real estate was incredible; it reminded me of my hours on the Xerox Alto, only better. Remember your first time using two large monitors? Yeah.

Munch was a young curmudgeon, and used the “Kong” monitor in the following way: As a set of 80 by 25 line terminal windows, all stacked up directly on top of each other. He’d have ten or twenty of these, and just switch between them. To anyone watching it was frustrating, but Munch knew exactly what he was doing, and where everything was.

“I can’t believe we gave you that monitor,” said his boss, “The way you’re using it, it’s just wrong.” But Munch was incredibly productive, and nobody really complained that he didn’t make use of the screen space the way the rest of us would have.

- – - -

The weather in Seattle was absolutely perfect for motorcycle riding, and I was out in it today, reflecting on all of this.

Munch’s real name was John Paulson, and he died of cancer in March.  I just found out a few hours ago. I miss him.

1. Keep quiet.
2. No, really: Shut the hell up. You cannot possibly improve things. Just let the train-wreck slide to a halt and enjoy the scenery.
3. Smile!

You have a large body of code that is using C just fine, thanks, and moving it to C++ (while probably “trivial” for some meaning of trivial involving a few weeks/months/years of refactoring) would expose you to every C++ yahoo’s whim.

“I wanna use maaaaaap! Why can’t I use map? Map map mapmaapmap.”

“Because this stuff runs in an interrupt handler, and is shared with thread-world code through a very carefully designed API involving a couple layers of synchronization. Many Bothans died to make this work well, and fast.”

“No map?”

“No map. Go read about NUMA.”

“What about streams?”

[buries head in hands]

tl;dr; It’s an idiot shield. Kids, get off the lawn

when you interview
please remember your basic data structures
like linked lists and trees
because i get pretty depressed
when i think about all the so-called
senior software engineers who don’t remember
this stuff or think it is all beneath them
i mean what have you been
doing all this time
writing linear searches
(that would explain a lot)
or architecting your way into a permanent
state of irreplaceable expertise
by erecting fucked up towers of xml and wsdl
schemas backed up by executable uml

oh god my head hurts
just thinking of the evil that people do
in the name of a little convenience

i once asked a candidate what kind
of structure he would use to do
pagination in a word processor

he said frankly i dont give a shit
so i thanked him for his time and
showed him the door
it’s probably the most honest response
i’ve ever heard

The heart is not located in the normal spot, in a PM. He’s back at work today, spreading joy around and asking for estimates.

“What can you get done in, say, the next ten minutes?”

“Huh?”

“There’s a status tracking meeting at four, and I need to report progress.”

“But we talked at ten this morning, and –”

“This is really high visibility, and we have to show we’ve done something since then.”

Ack.

He continues: “Or, we can give the project to Ralph. Ralph is good at reporting progress.”

“Ralph is a DBA. He doesn’t know which side of a curly-brace goes left or right.”

“But he gives good progress.”

“He thinks backslashes are the devil’s work. I’m not even sure he knows how to turn off his caps-lock key. Also, he doesn’t know C++.”

“You’re just being difficult.”

About five years ago I purchased The Bag from an old crone in a bad part of town. I’m not sure what’s in it, but a little bit of the stuff does wonders. I reach into my pocket, draw out a pinch and throw it in his eyes while muttering the charm.

He takes a step back, rubs his eyes and says “Ackblphltb?”

“You’re feeling on top of your game!”

“I am? I am!”

“You’re not being paid enough!”

“I’m so totally repressed and under-appreciated in this group. This sucks.”

“You’re going to interviewing, starting tomorrow!”

“I am so out of here. Maybe I’ll go to Google or Facebook.”

And that is how we do that.

The PM said, “This meeting was useful. Should we set up a recurring meeting on this subject?”

So I stabbed him in the heart.

I like to play a game sometimes, it goes like this: Take a current hot and screwed-up software methodology and try to project what its replacement is going to be.

For instance, the shiny bits of Scrum are starting to wear off like the cheap paint from a fake idol, but there’s still time to dust off the crystal ball and toss some coins in the air. And based on what I’ve seen killed dead on the altar of Agile (and it ain’t been pretty) here’s what I think will happen next.

Remember all those burndown and velocity tracking applications The Powers That Be bought for Scrum? They’re not going to go to waste: Get ready for micrometrics.  Practically everything a team does will be measured, analyzed and acted upon by project managers. Going over checkins and having a thousand eyes on things is going to be horse and buggy stuff. We’re going to see utter exposure of everything that goes into the head and out of the fingers of developers, every cotton pickin’ thing will be timestamped, correlated to features, and graphed during reviews (which will happen in real-time).

In the new world of micrometrical software, no sparrow will fall without a burrowcrat getting on someone’s case about it.

How long did it take you to fix that bug this morning? What source code did you call up? How many steps did you take in the debugger, and what did you look at? What unit tests did you actually write and run?  Did Joe, the dev next to you, fix a similar bug in less time, with better quality? How much email went back and forth about the issue? Do you often mis-spell keywords? Do you hit backspace a lot, or change your mind about function names or parameters? Does it appear that you know exactly what you’re doing?

What percentage of your keystrokes and mouse clicks and minutes of your day can be mapped to a product feature?

My guess is there are shops already doing this, where fractal attention to detail has become a recursive ass-covering game, and everyone has forgotten why they started writing software for a living.

Has it been that long? When was the last time you enjoyed making a hunk of code work?

Please, have fun every day.

My new favorite way to get rid of spiders in our house (they are welcome anywhere outside, but not in the house proper, and not in the garage) is to whooshify them with the shop vac. This is safe and efficient because with my superior engineering skills and some duct tape I’ve added an extension wand about yayyyy long to the existing extension wand’s extension. With my corrected vision I can spot a black widow spider from 20 feet away, and now that I have developed the capacity to whooshify them from a distance greater than their pouncing radius, life is better.

Except . . . I have this new irrational fear of vacuum cleaners.  Umgekehtervacuumenspiderenwhooshenobia is the phobia that a vacuum cleaner will experience a sudden polarity reversal and spray several years worth of very much alive and probably quite vexed poisonous spiders around the room. Hilarity ensues.

- – - -

In the software industry, the practice which is most like saving up poisonous spiders in a big bag for a later, less convenient time is the creative denial we call Bug Triage. It’s got other names, such as Managing Customer Expectations, Meeting the Quality Bar, or Getting the Fucking Project Shipped, but it all comes down to one thing: Stuffing defects into a bag (a “backlog” or similar purgatory) and getting the stupid collection of bits to paying customers without the bag or the customers suffering a catastrophic reversal of fortune.

The dirty truth of software is: It’s buggy. From the crappy hacks that run your toasters and flatscreen TVs to the expensive professional-grade stuff that keeps nuclear reactors from running amok and spewing out gigantic radioactive spiders, if something’s got a couple kilobytes of code in it, chances are it’s also got some pretty tasty bugs. We’re just not good enough as a species to put together a gadget with a few thousand parts without risking something blowing up.

Most of the time we don’t see them. Or if we do, they are minor annoyances — who cares if your smart toothbrush crashes?  However, occasionally there is a howler.  Some ship team’s pure, distilled denial lets something through that should have been an easily identifiable must-fix bug.

The truly pernicious bugs are the ones that don’t actually look like bugs. You think you’re happily slinging code around, and one day you wake up and realize that you’ve filed sixteen TFS reports in the last day, all about the same simple problem. You think harder and discover that other groups — people that you’ve never even heard of — have been holding daily war meetings about this bug. Six PMs and three managers are asking about status. (The actual issue? It’s a cosmetic thing affecting the color of a button in an error message, a one-character fix where a less-than should have been a greater-than).

The system has eaten itself, and you’re mired in a toxic, icky pool of process. You’ve become a living part of the bug. Your job? To keep the bug handling system alive by making more bugs. The bug-handling ecosystem has evolved to florish on the bug-making process, and if there aren’t enough bugs, it will make up some of its own.

My advice is: If it gets this bad, either get promoted so you can start lobbing hand-grenades into the machinery, or do something else.  Trust me, the system won’t miss you.