Managing Metrics Madness

The Black Belt proudly displayed a huge spreadsheet printout. “It’s the first time we’ve been able to see all our metrics in a single place,” he announced. “Everyone was surprised at how many we have.”

Including me. There were well over 100 metrics on the spreadsheet. I wondered if all of them were important. Even if they were, it would be beyond the ability of mere humans to understand the relationships among so many variables. Quite simply, there were too many metrics to manage.

Metrics are, of course, essential to Six Sigma. It’s no secret that we need metrics to manage our businesses; we can’t improve a process that we can’t measure. Most managers and leaders are trying to make sense of an overwhelming number of “important” performance measures. They usually have very limited success. What’s to be done?

The first step in corralling the metrics monster is to understand metrics. A metric is part of an operational definition. A definition isn’t complete until we’ve identified how we’ll measure what we’re defining.

It’s critical that the measurement is valid. This means that it measures what we think it measures, which is more difficult than you might think. Take something as simple as the diameter of a hole. Any experienced mechanical inspector will tell you that there are many answers to the question “What size is that hole?” Holes are not perfect cylinders. They’re not round on either end, and they aren’t the same size throughout their depth. The size we want to know depends on knowing why we’re interested in measuring the hole in the first place. Do we need minimum or maximum clearance? A press fit? Access through the hole? The validity of the hole-size metric depends on our answer.

A second important property of a metric is that it be reliable. Perfectly reliable metrics would always give the same result when what’s being measured is the same, even when assessed by different people or at different points in time. In some cases it’s possible to quantify unreliability, such as with a formal measurement systems analysis. In other cases, such as a customer experience or a destructive measurement, this is problematic. Despite the challenges, thought should always be given to making the metric as reliable as is economically possible.

Critical to Quality Metrics

Not all metrics are created equal. In nearly all cases there are a critical few metrics that can be used to characterize an outcome or an important determinant of an outcome. These critical-to-quality metrics, or CTQs, can be identified using subject-matter expertise, logic, common sense or observation. Their importance can be assessed using statistical methods ranging from a simple Pareto analysis to complex components of variance analysis.

Sometimes it’s not clear which of the many metric candidates are critical and, despite your best efforts, you find yourself with dozens of measurements. Don’t despair. It’s possible to reduce the number of measures you’ll focus on to a manageable few by considering the interrelationships among the metrics. If X₁ and X₂ have a correlation coefficient of 0.99, then knowing X₁ means that you know X₂. Why track both? Track the one that’s easiest or cheapest to obtain.

You can also use more advanced statistical methods to find the critical few. Factor analysis or principal components analysis tells you how to reduce a large number of variables to a smaller number of factors. These factors, in a sense, explain the variation of the variables. The factors should bear a logical relationship to the variables and make sense. For example, a customer survey might ask questions such as:

• Was the product what you expected?
• Was the invoice price the same as the quoted price?
• Was the delivery date the same as what the sales agent stated?

If statistical analysis indicated that responses to all these questions were related, then it might be possible to treat all as measurements of a single construct that might be called, say, “honesty.” The honesty score could be monitored on a dashboard instead of the three separate questions.
There are a limited number of reasons for being interested in any metric. I suggest that metrics are only of interest if they have one or more of these properties:

• They’re an important outcome
• They’re an input that affects an important outcome
• They measure something you can control or something that can be compensated for by something you can control

This implies that we have a solid model of cause and effect. We know that a given change will produce the desired result because we understand the mechanism that makes this happen. In other words, we have transfer functions linking inputs to outcomes. Many transfer functions are a result of knowing the subject matter, such as engineering or marketing know-how. Many others remain to be discovered by applying Six Sigma.

Are you interested in Six Sigma Training or have more questions on managing metrics? Contact Pyzdek Institute today!

Reproduction allowed if no changes are made to content and you link to this article.