Irony

Ever worry when you get onto an airplane? I suppose it's only natural to be slightly afraid when you get into a metal cage with 20,000 moving parts and miles of electric wiring, carrying several hundred gallons of highly explosive fuel 5 miles above the earth at speeds normally reserved for meteors. How do you reassure yourself? "They do all sorts of testing on these things to make sure they're safe." Which is, of course, correct.

Cars, trains, airplanes, nuclear power plants, chemical factories and the like all have one thing in common. They have extensive safety testing done on them to make sure that the chances of them killing everyone around is relatively slim. But can they really accurately test every part to see how long it takes to fail and, therefore, determine when it should be replaced? Of course not. It takes thousands, if not millions of hours for some parts to fail. They can't possibly run a whole group to failure to determine an average. Instead they'll just use the data they get from the part manufacturer or the sub-system contractor in a calculation for the failure rate of the Device of Death they're building.

But where do those failure numbers come from? One technique is called "Reliability Prediction of Electronic Parts". There are several different methods, or standards, used to determine what the failure rate of an electronic part is. Two of the most common are the Mil-217 standard and the Telcordia SR-332 method. These methods involve taking various bits of data about a component, such as its operating temperature, applied voltage, quality, etc. These bits of data are known as "pi factors". These pi factors are then multiplied against a base failure rate for the type of electronic component (capacitor, integrated circuit, transistor, etc.) to calculate an average failure rate for that particular component in that particular usage. So if you trust the US Military (which never makes mistakes, right?) or Telcordia Technologies (whoever the hell they are), then you pretty much know about how long it takes for your component to fail.

But most companies are still too lazy exert even this much work. That's where software applications come in, such as the fine products written by the company I work for. They can easily calculate all these pi factors and failure rates within milliseconds, based on a few user inputs. So now we also have to trust the engineer who inputs the parameters into the software.

But being Americans, this is still too much work. When you have 10,000 components in a single system, you don't want to be bothered with inputting every single one and setting the operating parameters. No, that's far too much work. Instead, we want a library of parts from various manufacturers already defined in the software so all we have to do is provide the software with a list of all the part numbers of the parts we're using, then the software will look up those parts and import their pi factors. So now in order for our Killing Machine to work safely, we have to trust the library and the person who assembled it. And who assembles said library? Me.

The other day, I realized what an awesome power over life and death I hold in my hands. thousands of reliability engineers from hundreds of companies are relying on the parts data in the library I'm making. If I screw up and enter a part as a Thyristor when it should be a Transistor, the consequences could be dire.

Take, for instance, the bug I found this morning in the code that imports a list of manufacturer's parts data to our library. For a certain capacitor, if it has a rated temperature of 65C or lower, the output to the library should be "CARTCR=RT65". This way the program will read the rated temperature properly and everyone lives. But due to a typo that I made (probably because I was reliving my previous nights adventures in the land of Azeroth in my mind), this was importing as "CARTCR=65". What's the big deal? Well, the next step down the road, the reliability engineer needs to use this part in his Suicide-o-nator. He tries to import the part, and the program gives him the error message "Part THX1138, invalid attribute 'CARTCR'. Setting to default." The engineer doesn't stop to think twice. He has tens of thousands of parts he just imported at the touch of a button. He doesn't have time to hunt through his entire project to find this part, nor does he really care, because he has no freaking clue what "CARTCR" even means or stands for. But (and this is where things get ugly) all hell breaks loose. The default rated temperature for this particular capacitor category is 85C. So in this engineers system, suppose he runs it at 75C. The software sees no problem with this, as the operating temperature is well beneath the rated temperature. The software calculates the part has a failure rate of 1 failure per 3 million operating hours. But such is not the case. For in reality, this capacitor is only rated to 65C and will explode at temperatures of more than 70C. In reality, the part failure rate is more like 3 million failures per 1 operating hour.

So the engineer signs off on his system, with an overall failure of once per solar cycle, thinking everything is safe. The maintenance engineer, seeing this overly optimistic failure rate establishes that, as preventive maintenance, the system should be replaced once every quiggly-jiggly-illion years. And all is peaceful in the land. Until the part is actually used on the Gizmo-o-Death.

"Tower, this is United 334 requesting permission to land," the black box records. "You are cleared to land, runway 3," comes the response. "Hello folks," the pilot says over the PA. "We'll be touching down just inside five - OH MY GOSH! WHAT IS THA---" followed by static. The charred remains of the plane plummet to the ground, landing on the low-income housing tract on the approach path to the runway, killing thousands on the ground. All this because there was a faulty capacitor on the landing gear control system, run at a temperature too hot, which was not caught by the software because some 24-year-old kid was day dreaming and bored at work.

So the next time you get on an airplane, or in your car, or even use an electric can opener, stop a moment to think whose hands your life is in. It just might be in mine.

So then you ask, "but the title of this column is 'Irony'. Where's the irony?" The irony is that with my luck, I'll probably be sitting in seat 14A of that United Flight 334.

[Editor's note: this article was edited, by popular demand, to make it less macabre. Which is ironic, because it's still pretty darn macabre.]