“When you’re developing metrics, you need positive, open communication with customers, suppliers, leadership teams and process experts. To help you get started on the right path, here are eight characteristics of a good metric:
- Meaningful to the customer
- Simple, understandable, logical and repeatable
- Shows a trend
- Clearly defined
- Data that’s economical to collect
- Drives appropriate action
- Tells how organizational goals and objectives are being met through processes and tasks”
I am retired now, but as a systems engineering on intercontinental ballistic missiles, a typical concern of mine (and many others) was to continually look at large amounts of data to draw conclusions on the readiness of the weapon system to meet its mission. The mission of intercontinental ballistic missiles is to deter war through readiness, and if deterrence fails, to take out the enemy’s military capability.
I needed to understand, is the reliability improving or decreasing? Is accuracy remaining adequate? Will the missile still fulfill its mission under the effects of nuclear war? Fundamentally, were the readiness metrics trying to tell me something?
An important corollary question was always: “Is this a real trend, or is there something wrong with the data or how I am looking at it?” Strict systems engineering disciplines and strong leadership made this process very effective. It can work quite well when done correctly.
In my situation, the mission was clear and constantly discussed. Analysts, especially the contractor, were incentivized for honesty, precision, and persistence. I was privileged to work within a metrics system that worked and worked very well.
How do you find the right metric?
The best general definition of metrics I have found has been from the Air War College lesson on “Quality in Daily Operations”, circa June 1995 textbook:
Unfortunately, there are too many examples of top leaders who fail to seek out good metrics that reflect their mission. Rather, they seem to seek out metrics that are simply manageable and meet certain “sub-optimal goals”, such as nice-looking annual reports or quick sound bites.
The classic case of poor use of metrics is “body counts” during the Viet Nam War. The 3 television networks were happy to report numbers of enemy people killed versus numbers of our people killed on a nightly basis (at dinner time). It was an easy metric to show to millions with little chance of misunderstanding. It serviced the government’s desire to show what looked like good news. It seemed to indicate always that victory was just around the corner.
But metrics must be closely coupled to the mission. With the mission in Viet Nam not entirely clear, this was always problematic. Was the goal to stem the tide of Communism? Reunite North and South under a South that was nominally “democratic” and capitalistic? Leave with honor? Do we even know today what the primary mission was back then?
What we got from the Vietnam War metrics was the dull drumbeat of fatalities in a mission that was hard for most nightly news watchers to comprehend. Actually, if the goal was to end the war no matter the consequences, perhaps body count was the perfect metric.
Good leaders look for the real goals and create real metrics. If the mission is unclear from the outset, that road is steeply uphill. But even with a clear goal and good metrics, there is another trap as exemplified by the current VA wait times scandal.
What if you cannot trust your data because you have somehow motivated your middle managers to “fudge” the data?
The best top leaders know how to anticipate the problem of bad data entry and create a system that avoids this, or at least detects it. The top leaders at the VA certainly had previous IG reports to give them strong hints of what was happening. And how could they not be interacting with their ultimate customers, the veterans, for anecdotal data?
So the moral of this story is, whatever you are in charge of, spend the time necessary to create, operate, and update your metrics and your analysis processes. This may even push you to clarify and better publicize your mission, which is always a good thing. Although the payoff is in the long run, it is well worth the time.