[follow-up] So in the end, resolving the issue we had with Comcast took someone who knew what they were doing, with good debugging skills and a good attitude, and about 15 minutes. It helps to be methodical. It’s fine to twist knobs and keep trying stuff, but you need to be scientific about things. Write stuff down if you need to. Be organized. Change one variable at a time.
1. Examine the data carefully
Clueless tech: “I don’t even know where to start.”
Good tech: “Hmmm, only some channels work when they ALL should. Let’s look up some of these channel frequencies on the channel map reference and see if they’re correct.”
2. See if you can state the problem simply (possibly by recognizing that there is a problem)
Clueless tech: “Lots of channels aren’t working. Huh. Did you give us money for them?”
Good tech: “Channel 647 isn’t working, and it should.”
3. Ask yourself if there’s any more data you can gather
Clueless tech: [Argues about subscription packages with the customer]
Good tech: “Hmmm, I have a signal strength meter with me, maybe I should measure this.”
4. For a complex system, try tracing a fault back to its source.
Clueless tech: [still arguing about the channels we’re paying to not get. Shows me a page on his phone that he says supports his argument that we shouldn’t get those channels, I do research in front of him to show that he’s flat wrong and looking at stuff from years ago]
Good tech: “Hmm, no signal here on that frequency, maybe sampling the signal closer to the street will help localize the fault?”
5. While it might make sense to twist a knob a few times to see if the problem just goes away, twisting the same knob thirty or forty times only makes you look like a frustrated skinner box rat when the researcher has gone on vacation and fogotten to fill the pellet jar.
Clueless tech: “Let’s do that thing that didn’t work before, again, a few more times, because it’s the only tool I’ve got.”
Good tech: [Already out the door and finding the cable closet]
Root cause was one of the two things I thought might be the issue. The first possibility was a bad channel assignment from the head-end (and believe me, the software that runs the head-ends is crap), the second was a cable trap that had been left on the line. Turned out to be the trap; a filter had been installed maybe ten years ago in a locked utility closet and never removed, and it was happily quashing the signals between 270 and 670 Mhz, and since Comcast went entirely digital in our area a few years ago, these traps should just have been removed.
The first tech just flailed and couldn’t actually think about the problem, but could have solved it if he had paid attention. The second tech knew a few failure modes, but more importantly he knew how to think about a problem.
6. Don’t design systems that lack end-to-end diagnostics. They will be expensive for you, and your customers will have little pity as they stand back and watch you flail. I’m happy to pay NASA to publically fail because they do stuff that is exciting, dangerous and hard; cable provisioning is none of those.