(An excerpt from the upcoming book: Fundamentals of Complex System Sustainment)
It’s amazing what you can see by looking.
In order to be prepared for your risk meeting, as discussed in the previous chapter, some amount of system observation must have occurred. In other words, the “O” of O-I-F.
Experts know that observation carries with it the potential to misunderstand or misinterpret. Preconceived notions of what the shadows on Plato’s cave wallshould mean could get in the way of understanding the truth behind the estimates. That is, after all readiness metrics are estimated for all parts of the weapon system, the system is still likely not fully understood.
For example, when a guidance engineer takes all the data from a test flight, one of the first steps is to make the best guess as to the actual trajectory. Even the most accurate global positioning data has enough error in time and position to affect any subsequent analysis. If the analyst doesn’t even know for sure where the missile was, how can any kind of accurate assessment of the missile accuracy be made? How can potential creeping accuracy degradations be found? This is accomplished by understanding potential observation errors and characterizing them to provide estimates of confidence in the observations. Small errors in observation can be tolerated. Larger errors call for more efforts.
This risk that the system remains incompletely understood is also mitigated by double-checking between different kinds of observations. This is a kind of “active observing”. If a set of factory battery tests predict 50% reliability for 10-year-old batteries, yet 9 missiles have flown with 10-year-old batteries with no failures, perhaps the factory tests need some improvements.
What should be obvious in all this is that good observations depend heavily upon excellent configuration tracking. How can you really know what your system is doing if you might have some errant component, perhaps an early production lot of gyroscopes, that consistently creates problems? Yet you don’t recognize the problem is limited to only those few gyros.
Yet another complexity must be laid upon the system sustainment function of “observing”. Your risk meeting participants will rightfully wonder if your observation has spotted a trend. Does the risk describe an emerging failure mode that will exponentially degrade the system over the next few years? Although widely regarded as impossible, the future must be predicted with some certainty.
Unchecked, all this becomes ridiculously expensive.
It was mentioned how serious a business it is to extract sustainment funding from decision-makers. This is appropriate, of course, as these top executives must be good guardians, typically of someone else’s wealth. So assessment must be efficacious, but it also must avoid using too much scarce funding.