Dissecting Significance: A Clear Perspective on Statistical Terms and Their Impact

There’s a significant amount of misunderstanding that arises when interpreting the term “significance” in the realm of statistical process control. For those in the world of quality and statistics, “significance” often carries a different implication compared to the interpretation understood by the general audience.

Consider this instance from my past experience where I was assisting a team of process engineers experimenting with a novel vapor-phase soldering process. This was a nascent process, thus a significant level of variation from numerous unidentified sources was present. Such statistical noise was strong enough to eclipse any signals caused by the controlled alterations we made in the variables under study. Upon observing the results of the experiments, I remarked that none of the effects seemed significant. One of the engineers disagreed, pointing out that according to established scientific principles, all the variables we tested were significant.

Interestingly, both our perspectives were accurate but originated from different interpretations of “significant.”

Statistical significance is a term commonly used by statisticians and quality engineers, typically implying significance tests like Student’s t-test for equality of means. In layman’s terms, significance testing is essentially asking, “Is there a genuine difference between these elements, or could the variance be attributed to sheer chance?”

The real challenge surfaces when these test results are shared with individuals outside the realm of statistics. In everyday language, “significance” and “importance” are considered synonymous. However, when statisticians discuss “statistical significance” (often omitting the “statistical”, thus intensifying the confusion), the terms “importance” and “significance” lose their correlation. In essence:

A difference or effect can be significant yet unimportant.
A difference or effect can be significant and important.
A difference or effect can be insignificant and unimportant.
A difference or effect can be insignificant but important.

In short, “important” and “significant” in a statistical context are not interchangeable. To illustrate, let’s consider the following examples:

Important but not significant – Imagine a hospital where the mortality rate for coronary artery bypass surgery doubles from one week to the next. The loss of life is undeniably important. However, if the average mortality rate is low (e.g., 0.5%) and the sample size is small (100 surgeries per month), the fatality rate can double from the previous month without being statistically significant.
Significant and important – Suppose a part’s average hole size falls below the lower control limit, causing it to malfunction in the field. Here, the alignment between significance and importance is purely coincidental, and its occurrence often leads to confusion in other situations.
Unimportant and not significant – A minor and statistically meaningless change in the part’s hole size that doesn’t affect its functionality is neither significant nor important. Again, the alignment between the two terms here often results in confusion.
Significant but not important – Carpenter A cuts wall studs to within ±1/64″ of the nominal length, while Carpenter B cuts them with double the variation at ±1/32″. The contractor, however, can tolerate a difference of ±1/8″. As both carpenters cut a large number of studs, even minor differences can be statistically significant. However, a difference of merely 1/64″ in the total spread of a sawing process holds no architectural or economic importance.

In the field of statistical science, “significance” is a technical term with a precise definition based on the statistical test being employed. Ironically, it’s possible to determine statistical significance without understanding the relevance of the numbers or the application of the results.

Conversely, “importance” is a non-technical term, inherently posing the questions: Important to whom? Important for what? Determining the importance of a particular number isn’t as straightforward as analyzing statistical significance and doesn’t offer easy, pre-determined answers.

We should practice thoughtful consideration before reacting to an observed difference, regardless of whether it’s significant.