Clownspace

What does a buffoon do when his project is utterly out of code space, running late, and rather horribly extremely buggy? Why, he adds more features, of course!

Customer: “We want a log of the temperature readings for the last week, in degrees Fahrenheit _and_ centigrade.”

Contractor-Buffoon Figby: “Okay!”

Customer: “A log of the last week’s current draw would be neat to have, too.”

Contractor Figby: “Yup, gotcha covered!”

Customer: “We’d like to have the following eighteen things done on the wire protocol so that our techs don’t have to upgrade their versions of LabView, because that costs money and you always say ‘Yes’ to us.”

Contractor Figby: “Yes! Sure! Whatever you want!”

The problem was that Figby the Contractor couldn’t program his way out of a paper bag. His code didn’t work (it was riddled with bugs he was in denial about), and he was out of room in his part’s code space by a factor of three (and that was just for the boot loader of the code he hadn’t written yet).

Figby didn’t know how to say “No” . . . so we said it for him.

We tried being gentle (“Why don’t you do it this way? You’d save a lot of space and it would be faster and more reliable”). We talked and filled up some whiteboards with reasonable designs.  No go; he was addicted to making his management happy by giving them — well, more accurately by PROMISING them — whatever they asked for.

We tried _not_ being gentle (“If you continue the way you’re going, you’re going to fail and get a lot of people upset at you”). This entered his ears and went into his brain and somehow got processed as “I need to add more features so I don’t fail.”

Figby had been hired into a group notoriously bad at hiring software people. We tried going through that other group’s management (“He’s going to fail”) and our own management (“That contractor and that management over there are all going to fail”). Nothing worked. We were just the usual high-strung software guys with our shorts in a twist about nothing important. If we’d hired a brass band and done a parade in the parking lot with a big float and Figby in it with garlands holding a banner saying “My code is too big and will be late and it won’t work anyway” we’d have been ignored just as all the other software engineers, always Cassandras, have been ignored from the very first big disaster, probably some spec work for Hammurabi involving prayers for rain that didn’t deliver on time.

Status: We had tried the talking thing, and failed. The buffoon was going to ruin our project, make our lives miserable, and pile the blame on us. It was time to get out the knives.

—-

A little technical background: The piece of code in question was for a small microcontroller. Not the smallest controller we could buy, but darn close: It could hold 4K of code, and it had comic relief quantities of RAM and some EEPROM. The part didn’t need to do very much, but what it had to do was critical; if its code didn’t work, the whole product would be a brick because this little fella was responsible for turning the unit’s power on and off at the right time.

To be safely field-updatable, two copies of the code had to fit in that 4K of flash so that two fully functional images would always be present. Figby the Contractor’s boot loader was 3K and thus well over budget, and not only wasn’t he done, he was still adding and promising new features while ignoring his backlog of bugs.  “Oh, don’t worry,” he said, “The code will just have to go on a diet.”  We were pretty sure that no amount of dieting was feasible.

We were ignored in meetings. We had trouble getting any of the hardware, much of which was reserved for testing (it was unclear what they were testing, with so little of the code working). The train wreck slid closer and closer to us, with Figby yanking incompetently at the controls and apparently still promising that things would be working fine. After a little dieting and bug fixing, of course.

By late November I’d had enough, and I snapped.

—-

How I Spent My Thanksgiving

On the Wednesday before Thanksgiving I went home in a fuggy depression. The next month wasn’t going to be fun. The months after that were not going to be fun, either, since we were the poor bastards who were going to have to make the thing work after Figby finished his contract and skated off to his next disaster.

No hardware? Why should a little thing like that stop a software guy?

Thursday morning I downloaded the specs for the microcontroller from the manufacturer’s public web site. Then my family drove to my in-law’s house, but I’m afraid I wasn’t very social. When we got there I found a comfy chair and wrote some perl to extract the opcode tables and assembler syntax from the chip’s documentation. I started hacking.  By early evening, after a Thanksgiving dinner that couldn’t beat, I had an emulator for the chip’s CPU running some test instructions.

The next day I got a bunch of the chip’s hardware registers emulated, as well as emulated RAM, EEPROM and Flash.

The toolchain vendor had license-protected the C compiler so I couldn’t use it on my laptop at home, but the assembler ran just fine (otherwise I would have written one; probably would have taken three or four hours with the tables I’d already made). So on Saturday I wrote the boot code and got the emulator to run the code and print “Hello, world!” out a simulated serial port.

On Monday I spent an hour or so getting the code to run on the real hardware (I spent half an hour of this sneaking over to the hardware group to “permanently borrow” some hardware and a JTAG probe from one of the labs they’d forgotten I’d been given access to on a prior project). The code didn’t work the first time, but after I set up the baud rate correctly, the second time it ran it printed out “Hello, world!”.  Carriage return. Motherfucking linefeed.  Yeah.

Then I told my boss what I’d done.

“Don’t work on this,” He said. “It’s going to be a big problem and cause a lot of friction.” He winked hard. I think he gave himself an injury making that wink.

“Yeah,” I said, [WINK], “But it’s going to save the project.”

“I know, but it’s going to cause trouble.”

So on Tuesday he called me to his office and said, “I talked to some people. Go go go go, finish that up. Just don’t say anything. We’ll work it out.”

And I wrote the rest of the code in C and assembly in a week or two, redoing what had taken the contractor four or five months.

—-

My cow-orker Nate was jealous. “You’re having too much fun,” he accused.

“I am.” In fact, I did feel bad about it. I was having a grand time writing low-level code in C and assembly, and Nate was working on something in C++ that was definitely not enjoyable.

I checked in some code one evening. The next morning Nate came up to me and said, “Sorry, I couldn’t stay away. I fixed a bug in one of your routines last night.”

“Okay,” I said.

“Then I ran out of space and had to re-code some other routines in assembly to get enough bytes back.”

“Sure, feel free.”

“Holy crap, this is fun.”

And with that, we were a team. It went back and forth like that for a while. I would fix a bug, or Nate would, and we’d need to scavenge bytes by recoding the somewhat fat code that the C compiler emitted, and we’d often wind up with more bytes free than we’d started with. I showed Nate how to multiply by ten, a hundred, and one thousand using a little inline recursion trick, and he called me a sick fuck. Nate rewrote the little build system I’d done and saved even more bytes. It became a contest.

We wound up with six bytes to spare, and Nate said he felt like he’d been on a two year course of anti-depressants.

A few months later we had a last bug fix and had to scavenge more space.  We wound up with six bytes free again after the changes, and those bytes were pretty tough to find.

A manager came up to me a few days later and asked for a feature.

“Nope,” I said, “We’re out of bytes.”

“But it’s a bug.” (I guess the lack of that feature was a bug to him).

“We fixed the last bug on Tuesday. We don’t have space to fix any more. We’re done.”

And that was that. It was nice to be done.

—-

For the entire project, Nate and I never used a debugger; we just did a little strategic output on the serial port. We could have used a JTAG debugger, but after a while it was just easier not to bother.

—-

What happened to Figby?

About a month after we got our version of the firmware working, the other group told him he wasn’t needed. That’s the way it is with contractors. ->Poof!<-

His code, utterly useless, was simply deleted from the repository. Although we would probably have preferred to nail it against the wall, as a warning to future contractors.

“What’s that hard drive doing up there, with the spike through it?”

“Let me tell you a story….”

—-

And the morals of the story are:

1. Crappy and entrenched workers are a fact of life in most big organizations. They can’t necessarily be counted on to spontaneously self-immolate, so you have to be prepared to end-run them.

2. When you do that, get management buy-in at the right time. Like, when you have things largely working and you’ve dealt with (or at least can enumerate) most of the risk.

3. Remember to appear humble and apologetic as you crush their spirits.

4. When you see a software guy walking down the hall holding an oscilloscope, smile.

This entry was posted in Uncategorized. Bookmark the permalink.

18 Responses to Clownspace

  1. Ingo Karkat says:

    Great story. What happened to Figby (and his code)?

  2. Andy Konecny says:

    If a harddrive has a big spike through it, how could you tell if it has Figby’s code on it or not? Ideally any drive he used or even shared space with, would do to make such a powerful talisman.

  3. Tom says:

    A fine time had by all. Especially Figby, who (*SPOILER ALERT for next posting*) cheerfully added all this experience to his C.V. and just a short week later aced the interview with the H.R. team of the same company – they were fed up with hearing all these complaints about bad hires so reasoned that acquiring a real coal-face trooper with prior experience in the same field as the new Software Hiring Director would fix it forever.

    At least there’s a silver lining: we’ve long known that “A’s hire A’s, B’s hire C’s” but at last we can learn: who do the C’s hire? and just where do they find them!?

  4. Dan Weatherman says:

    Moral # 3 is simply sublime ;
    “Remember to appear humble and apologetic as you crush their spirits.”

    Thanks for sharing! These posts are always great fun to read.

  5. Kevin Clark says:

    Oh yeah! A skunkworks project to get around the total incompetence of a co-worker (and…more importantly…to SAVE THE PROJECT!). Indeed — these sorts of projects are frequently like taking “a two year course of anti-depressants”.

    Great story!

  6. Adam says:

    The software guy with the oscilloscope is probably smiling too. I know I was when I was debugging an audio routine on a microprocessor.

  7. Great story. A really fun read. I had a similar experience where in my first programming job I discovered that our main website was being held hostage by the original programmers by saying “we used proprietary libraries, so we cannot release the code to you”. I got my hands on the ftp creds, dumped the whole site overnight and started cranking way on the perl. My boss said “do NOT work on this site. If my boss learns that our team can fix it, he’ll make it our responsibility”. I couldn’t NOT work on it. I worked on that site over Christmas break and I still hold this experience as one of my fondest professional memories. I was able to eventually get it running, move it from Oracle to MSSQL and launch it on our servers. The outcome doesn’t really matter, I agree that the way these projects affect us cannot be described. 1 year later, I was managing the team and we needed to rescue a rocky pivotal implementation. I told my newest intern “We need to fix this implementation, but we officially can’t be involved.” I had no idea at the time that I put him in the same situation I had been in. He did it, worked evenings, weekends, etc. He would call me at 11 at night and say “hey, go to this IP and check out what I’ve done”. He had a blast. Why can’t we figure out a way to make all projects work like this?

  8. Donovan says:

    Love it. Worked @ BBY, similar stories. One thing of note, these days they *never* let contracts expire if the contractor is good. Anyone coming out of there who’s contracted ended, run-away.

  9. Azgarn says:

    After reading your story, I’m fairly certain Figby now works as an MSSQL admin at one of my company’s client firms. Save me.

  10. Kevin says:

    This is so frickin’ awesome. I remember my first software job — writing board-level assembly and ‘C’ for a telephone switch. My debugging tools were an oscilloscope, an electronics technician, a dumb terminal and keyboard.

    It seemed primitive at the time but looking back now it was glorious.

  11. hawkse says:

    Great story! I suppose this wasn’t one of those where Figby walks away with all the credit. What you did works in some situations, in other situations, you just have to sit back and let others fail, even if you can see the trainwreck coming far, far way ahead.

  12. Xelous says:

    I think that Figby son-of-a-bitch sits thirteen feet from me….

  13. Anonymous says:

    Funny read. Not so funny in RL, third or fourth time in row…

  14. David P says:

    Figby is multi-talented; I’ve seen him consult on non-IT projects as well.

    I once told my boss that Figby’s work was crap and needed to be redone from the bottom-up. That was in October, and my boss decided that were too pressed for time to fix it.

    Come February, the work was still crap, but suddenly fixing things became my job.

    “Overtime? You’re on salary, not hourly.”

    Gee, thanks.

  15. bill says:

    that is absolutely beautiful.

  16. Pete says:

    About 10 years ago I was working for a company that produced some software for an HD64180 based board.

    They used a really old MS-DOS based commercial Z80 assembler from around 1983. It just about ran in the Windows XP command prompt, but had several shortcomings:

    * Being so old, it only supported 8.3 filenames
    * It had a lot of cryptic and bizarre error messages
    * If you referenced too many external symbols, or marked too many public, it would fail. But the number of symbols wasn’t fixed. After years of practically superstitious rituals to alleviate this problem, we eventually tracked it down. The number of externs you could use depended on how many environment variables were defined in your shell (!)

    there were many other issues as well, but those were the main ones.

    The software manager refused to let us move to another assembler, because the one we were using had its own whacky syntax (not just for the sources, it even had its own make-like system) and he didn’t want to invest the time converting our codebase.

    So, in my spare time, I made my own syntax compatible assembler, linker and make-like system compiled directly for win32. Not only did it have proper error messages (I used GCC format so that you could just double click them in Editplus), it also didn’t crash, and could handle as many exported or imported symbols as you have available memory for. It ran several times faster. It generated byte-for-byte identical output on all of our existing projects. It assembled HD64180/Z180 opcodes directly, and even natively supported the memory banking scheme of our board.

    The software manager didn’t let us use it.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>