...as I talk about one of the most boring subjects known to man.
For those of you with persistent insomnia, a macabre fascination with what I've been doing at work over the last few years, or my adoring fan who just have to know what I'm up to (hi mom!), read on.
Firstly, point your web browsing thingamabobber to (I'll give this in the phonetic): India Sierra Oscar Golf Romeo Alpha Papa Hotel (dash) Sierra Oscar Foxtrot Tango Whiskey Alpha Romeo Echo (dot com). If you entered it correctly, you will be greeted with a smattering of words—or as I like to call it, a smwordening. You may close the smwordening (or if you're really interested, you can click to learn more).
 |
| No, you will not understand what half these terms mean. |
Once you're into the website proper, click the "Downloads" link on the right side of the menu that runs across the top of the page. To continue, you will need to fill out a download form with your name, company, address, and the time that you leave for work. If you don't want to fill out this form with your own personal information, just do what I do:
 |
| Ask for Jenny. |
Or you can just put that you're Joe's mom and you want to see what he's been hard at work on.
Where it asks for your email address, be sure to use the email address of some poor sap who you don't know, but who has a convenient-to-type address. Or just something random and funny. I recommend using joecool(at)gmail(dot)com.
Once inside the download area, select the first "Download" button.
Then wait. And wait. (Our FTP server is kinda slow.)
After the hamsters have personally delivered the requisite bits to your ISP, you can run the installer. Psych! No, you can't. You need the password to install! Alas and alack, that is not something I can post here, even in the phonetic. To get the password for the demo installer, you will need to contact me via email. I can be reached at something_random_and_funny@gmail.com, or some.poor.sap@yahoo.com.
Once I've given you the password—which isn't "password", "swordfish", or "xyzzy"—you can install Reliability Workbench. It requires the .NET framework 4.0, which will be installed if you don't already have it. When it's done installing, run the program. The "demonstration mode" dialog will appear. Click "Continue in demonstration mode".
OK, now you're ready to see what I've been doing for the last few centimeters of hairline. If you've followed me up to this point, you should see something that looks like this:
 |
| Rollover to see the UI terminology. |
Choose "Reliability Growth" from the module selector.
The Reliability Growth module is my baby. Even before Damien was born, I was hard at work nurturing it and developing it into something worthwhile. My boss basically said. "We need a Reliability Growth module. Here's a document that explains what that is. Here's the framework for the program. Go." I did all the coding, UI design, testing, and documentation.
But what is Reliability Growth, you ask? Excellent question! Reliability Growth is an analysis method used to determine if the reliability of a system is increasing, decreasing, or staying about the same over time. It can be used to examine the effectiveness of design modifications or changes in a maintenance policy. For instance, by recommending oil changes every 3000 miles vs. every 5000 miles, does the engine reliability of a fleet of vehicles increase? By using a different weight of oil, is their an appreciable impact on performance?
There are three basic Reliability Growth methods that our software uses: Power Law, Power Law for grouped failures, and the Crow Discrete method.
The first method, the Power Law model, is suitable for continuously-operating systems with immediately-revealed failures. Data consists of the time to each failure. The software will analyze the times to failure to look for an increasing or decreasing trend. It also provides a statistical test for growth, as well as a goodness-of-fit test that verifies the predicted growth curve matches the data.
To begin, select the Add, Growth Set menu option, or corresponding toolbar button. The Growth Set Properties dialog will appear. Choose the "Items" tab and enter the following data:
Click OK when done.
Let's pretend that we tracked the failures of our disgronificator over time. We want to know if our new maintenance policy is having a statistically significant impact on its reliability. We recorded the time at which each failure occurred, and so we enter them into the Growth module.
If it is not already shown, select "Plot" from the Right Window Mode selector. The program will show a plot of the cumulative number of failures vs. time.
If it is not already selected, choose the "Power Law" option from the Method selection drop down list on the toolbar, and select the Analysis, Calculate Method Parameters Automatically menu option or corresponding toolbar button (big red Eta on the toolbar).
The program will automatically calculate values λ and β such that:
E[N(T)] = λTβ
Where E[N(T)] is the expected number of failures to have elapsed by time T. These parameters describe the best-fit growth curve, and are displayed to the right of the plot. Also, the curve described by λ and β is displayed on the plot.
λ is the scale parameter. Ignore it. β is the one we want. This is the shape parameters. β > 1 indicates negative reliability growth (or the reliability is getting worse), β = 1 means zero growth (neither improving nor worsening reliability), β < 1 indicates positive growth.
Next is the U statistic. I won't go into details on how it is calculated, but you can look in the Help file if you're interested. Under the hypothesis that there is zero growth, U will be a standard normally-distributed random variable. By comparing U to the critical values of the standard normal distribution, the Growth module can determine if there is evidence of positive or negative growth. A two-sided test at the α significance level has critical values u1–α/2 and –u1–α/2. If U falls between these two values, we can say that U is a standard normal random variable at the α significance level, and that there is no evidence of reliability growth. If U < –u1-α/2 or U > u1-α/2 then there is evidence at the α significance level for positive or negative reliability growth, respectively.
Don't pretend you understood any of that, unless your screen name starts with "M" and ends in "r. Pickles".
You can change the significance level in the Growth Set Properties dialog. Double-click "GS1" in the project tree on the left; select "GS1" and choose the Edit, Properties menu option or corresponding toolbar button; or select "GS1", right-click it, and choose "Edit Properties" from the context menu. Either way. I'm not picky.
You can choose the significance level on the Analysis Options tab.
The program will show the calculated U statistic, the standard normal critical value for the chosen significance level, and a line indicating whether there is positive, negative, or no growth.
The program will also perform a goodness-of-fit test to determine if the power law model adequately fits the supplied data. This is done by calculating a Cramér-von Mises statistic and comparing it to the Cramér-von Mises critical value for the 10% significance level. If the statistic is less than the critical value, then the test is passed and the curve fits the data.
I don't pretend to understand how that one works.
Lastly, the program will also calculate the 90% upper and lower confidence bounds for β, as in "I'm 90% confident that I don't understand a lick of this, more or less."
If you want to see the reliability trend, you can also view graphs of the failure intensity and the MTTF (mean time to failure) over time. Select one of those plot types from the Plot Type selector above the graph.
 |
| Plot Type selector |
 |  |
Next up is the Power Law model for grouped failures, also known as case two of the power law. Like the regular Power Law model, this is used for continuously-operating systems, but differs in that the failure times are not known exactly. Instead, data consists of discrete time intervals, and at the end of each interval is a known number of failures.
Add a new Growth Set (Add, Growth Set menu item, if you've forgotten), and enter the following data:
Click OK when done. Pretend this data represents the hyper-electronic transducent geoplasmic nodal collectimizer component of our disgronificators. If that component fails, we don't know about it until we open up the disgronificator and check. Every 200 hours we check all our disgronificators and see how many have failed. We make some repairs and continue. We want to know if the changes to our repair process are improving reliability, so we enter the number of failures that each inspection detects, and let the software figure it out for us.
Select the "Power Law, Grouped Failures" option from the Method selector drop down list. As before, the program will calculate λ and β, test for growth and goodness of fit, and give upper and lower confidence bounds. However, here the test for growth and goodness of fit are both based on the chi-squared distribution, which in itself was a bit of a challenge. It was tricky finding an algorithm that would calculate chi-squared critical values for an arbitrary number of degrees of freedom. In the end, I found an old open-source math library for C that I adapted. Nothin' like debugging 20-year-old uncommented code written by Some Guy on the Internet.
The last method is an adaptation of the power law for discrete systems, known as the Crow Discrete model, for its developer, Dr. Larry H. Crow. Now, let me tell you something about this method. I curse the day Dr. Crow was born. The grief he and his discrete power law method has caused me is greater than that of Andromache felt for her beloved Hector. Allow me to explain.
Like the Power Law model, the Crow Discrete method finds λ and β such that E[N(T)] = λTβ. However, this method is suitable for one-shot systems, such as a rocket launch, which are either successful or failures. Testing takes place over successive stages. Each stage consists of several trials, each of which is either successful or not. After each stage, a design change is made that will impact reliability. We want to see if there's an increasing trend in reliability.
No problem! says Dr. Crow. That's easy! The answer will be the values of λ and β that satisfy these equations:
 |
| Yeah, have fun with that. |
t is the cumulative number of trials through stage i,
ni is the number of trials in stage i, and
mi is the number of failures in stage i.
"AAARRRGGGH!" says I. Of course, nowhere did Dr. Crow give any advice on how to solve these equations.
To put this problem graphically, if you plot the sum of the absolute values of the two equations for a range of λ and β, and invert the graph, you get this:
 |
| λ = 0.02 to 2.0, β = 0.01 to 1.0 |
Your mission is to algorithmically find the highest peak of that graph. Oh, but it's not even so simple as try a range of λ and β, find the highest peak and iterate towards the solution from there. Oh, no no no. Those peaks have fractal-like complexity. It's most likely you'll just hit the side of a mountain and discount it as having too low a value.
 |
| λ = 0.565 to 0.664, β = 0.73 to 0.83 |
After Googling around, I stumbled upon a paper that suggested a genetic algorithm approach. I followed that, implemented it, and hey! it worked! Kind of. Sort of. Well, so long as I didn't blow on it too hard, it sometimes gave me the correct answer for one particular sample data set. But most of the rest of the time it was hilariously bad. And slow.
But it was cool! It taught me evolutionary programming and genetic algorithms and awesome buzzwords. I was so sad when I had to remove it from my code. I left a haiku in the comments:
A work of genius,
This code was simply awesome,
But superseded.
Anyways, I stumbled along with that method for a while, right up until final beta testing when I realized I would be ashamed if we charged customers for that module. So, with the clock running down until final release, I turned back to the internets.
After some searching, I found an obscure DOD document site that sold a paper that, while first researching the problem, I had only heard of in songs and stories. The paper is called "Maximum Likelihood Estimates for the Discrete Application of the AMSAA (Army Material Systems Analysis Activity) Growth Model", by William P. Clay, written in 1987. (AMSAA is the organization Dr. Crow was working for when he invented the Crow Discrete reliability growth model.) I bless the day Mr. Clay was born.
You see, William Clay not only described an algorithm to solve the Crow model (I didn't completely understand it; my eyes glazed over when he started talking about iteratively solving the double-partial derivatives to find the maximum likelihood estimators), but he provided the code for a computer program written in FORTRAN 77! Let me tell you, never have I been so happy to have taken a semester of FORTRAN in college. "Never use it," my left kidney.
So anyways, after converting a FORTRAN 77 program to C#, for which I think I insulted my compiler for not being object-oriented enough, I got it to run and solve the problem.
So you can appreciate all my hard work, add a new Growth Set with this data:
So suppose we were testing nuclear bombs by dropping them on Moscow. In the first 14 attempts, 5 failed to explode. We made a design change and then tried again. In the next 19 attempts (trials 15-33), 3 failed to explode. After another change, we made 15 more trials (34-48) and had 4 failures. After the last change, we had 20 trials (49-68) and 4 failures.
Click OK, and choose "Crow Discrete" from the Growth method selector.
We now see that we are more reliably nuking Russia.
Don't say I never do anything for you.
