Comcastic (The Tale of the Good Tech)

[follow-up] So in the end, resolving the issue we had with Comcast took someone who knew what they were doing, with good debugging skills and a good attitude, and about 15 minutes. It helps to be methodical. It’s fine to twist knobs and keep trying stuff, but you need to be scientific about things. Write stuff down if you need to. Be organized. Change one variable at a time.

1. Examine the data carefully

Clueless tech: “I don’t even know where to start.”

Good tech: “Hmmm, only some channels work when they ALL should. Let’s look up some of these channel frequencies on the channel map reference and see if they’re correct.”

2. See if you can state the problem simply (possibly by recognizing that there is a problem)

Clueless tech: “Lots of channels aren’t working. Huh. Did you give us money for them?”

Good tech: “Channel 647 isn’t working, and it should.”

3. Ask yourself if there’s any more data you can gather

Clueless tech: [Argues about subscription packages with the customer]

Good tech: “Hmmm, I have a signal strength meter with me, maybe I should measure this.”

4. For a complex system, try tracing a fault back to its source.

Clueless tech: [still arguing about the channels we’re paying to not get. Shows me a page on his phone that he says supports his argument that we shouldn’t get those channels, I do research in front of him to show that he’s flat wrong and looking at stuff from years ago]

Good tech: “Hmm, no signal here on that frequency, maybe sampling the signal closer to the street will help localize the fault?”

5. While it might make sense to twist a knob a few times to see if the problem just goes away, twisting the same knob thirty or forty times only makes you look like a frustrated skinner box rat when the researcher has gone on vacation and fogotten to fill the pellet jar.

Clueless tech: “Let’s do that thing that didn’t work before, again, a few more times, because it’s the only tool I’ve got.”

Good tech: [Already out the door and finding the cable closet]

Root cause was one of the two things I thought might be the issue. The first possibility was a bad channel assignment from the head-end (and believe me, the software that runs the head-ends is crap), the second was a cable trap that had been left on the line. Turned out to be the trap; a filter had been installed maybe ten years ago in a locked utility closet and never removed, and it was happily quashing the signals between 270 and 670 Mhz, and since Comcast went entirely digital in our area a few years ago, these traps should just have been removed.

The first tech just flailed and couldn’t actually think about the problem, but could have solved it if he had paid attention.  The second tech knew a few failure modes, but more importantly he knew how to think about a problem.

6. Don’t design systems that lack end-to-end diagnostics. They will be expensive for you, and your customers will have little pity as they stand back and watch you flail. I’m happy to pay NASA to publically fail because they do stuff that is exciting, dangerous and hard; cable provisioning is none of those.

This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to Comcastic (The Tale of the Good Tech)

  1. C'est moi says:

    Change one variable at a time.

    Anybody who knows anything about experimental design knows this is less than optimal advice. It’s the programmer’s equivalent to naive physics. I have to admit that being a programmer I instinctively do the “change one thing at a time” too. I only wish I had the motivation to read up on experimental design.

  2. Tomsci says:

    It may be an oversimplified assumption to assume that changing any one thing will fix a problem, but attempting to change multiple things at once while trying to track a problem down (before you fully understand what the problem is) makes it far harder to figure out what’s going on.

    It works if you are doing something noddy like a bisect between two source control revisions looking for the point at which a regression was introduced (and you have a known good regression test case), but in the general case? It’s a complete nightmare. By all means see what happens if you change X, Y and Z, but before you do it’s worth checking what happens when you change any one of X, Y or Z. For example if X breaks the system even worse, it’s better to check for that first rather than merely knowing that “hmm, one of X, Y, or Z, or one of the interactions XY, XZ, YZ or XYZ made things worse”. Some problems are hard enough without purposely increasing the number of things you’re not sure about.

    There is a place for knob-twiddling it’s true, sometimes the only way to get a handle on a problem is to fiddle with the system until you get a ‘feel’ for what’s going wrong. But that’s mainly something you resort to when the problem is intermittent or difficult to reproduce consistantly. In this case, the problem is consistent and reproducable so the good tech should be able to leap straight in to the step-by-step

    To summarise, twiddling knobs = bad, thoughtful linear progression accumulating data one step at a time = good.

Comments are closed.