This week’s post is an excerpt from the paper I am presenting at AIAA’s SPACE 2017:
This paper provides a 5-step approach for an effective and affordable assessment program.
- Use your “free” data
- Look to your repair depots
- Set up an age surveillance program
- Establish processes for special testing
- Analyze your data to create information
Analyze Your Data to Create Information
For analysis, the best approaches to effective and economical observation are focused on to doing it right the first time. If not, reporting the resulting information to your supervisors and decision-makers turns into multiple embarrassing training sessions. Areas most likely to go wrong are your inability to convey the trend, why the trend is important, why you think it is real, and why it needs to be identified immediately or someday soon as a risk.
Many people with many skills are needed to provide effective and economical observation. Three roles must be done well to succeed in analysis and reporting: the statistician, the engineer, and the person chosen to communicate with the sustainment organization. The statistician must be competent and comfortable with a wide range of options for modeling what the engineer believes is happening. A competent statistician will balk at the idea of using mathematics trends to forecast the future. The engineer must have the competence to observe the system and conclude what wear or age process is at work to create the symptoms observed. What new emerging failure mode is this? Together, they can create the model that displays the trend and the engineer can explain why and in what manner it would continue into the future. The communicator must be able to relay all this, especially to the decision-makers, along with why the particular mathematical model is best in this instance.
Good decision-makers know that no model is perfect, systems are hard to observe, and mistakes are easily stumbled into. Reporting needs to convey an understanding of this uncertainty and what, if anything, can be done to decrease it.
One method is double-checking between different kinds of observations. If a set of factory battery tests predict 50% reliability for 10-year-old batteries, yet 9 rockets have flown with 10-year-old batteries with no failures, perhaps the factory tests need some improvements.
Good observations depend heavily upon excellent configuration tracking. How can you really know what your system is doing if you might have some errant component, perhaps an early production lot of gyroscopes, that consistently creates problems? Yet you don’t recognize the problem is limited to only those few gyros.
Other sources of error include poor test plans, ineffective test directors, changing test conditions from year to year, more than one emerging failure mode confused as one, and locked-in thinking that uses data to confirm a strongly held belief.
Some errors are baked into the process. So, a good sustainment organization is always looking for ways to improve its processes. Your system will be constantly changing over time as new failure modes emerge, operators come to expect capabilities you didn’t expect, and the world changes around you. Your organization’s processes must keep up.
Step one in this activity is to periodically recheck your assessment program to see if it is looking at your entire system, looking at sufficient subsystems to ensure lead times, learning from new techniques from other assessment programs and doing all of this across the spectrum of the readiness factors that your operators or warfighters need to complete their mission.