The second phase in Six Sigma’s DMAIC framework is “Measure.” During this phase the measurement system is evaluated to assure that the data that will be used to make decisions are valid. I.e., they consistently measure what they are intended to measure. This is crucial to decision making. In the current COVID-19 pandemic it is even more important because test data are used to disburse billions of dollars and to inform decisions about locking down or reopening businesses and schools and other places where people gather. Millions of lives and businesses are impacted by COVID-19 decisions.
Many medical and policy decisions about COVID-19 are based on testing. Primarily antibody testing. Antibody tests are an important public health tool to identify individuals with previous COVID-19 disease. This enables assessment of the spread of infection and the need for public health interventions. The Cochrane review 1 summarizes research evidence available up until the end of April 2020 to see whether antibody tests:
- are accurate enough to diagnose disease in people with or without symptoms of COVID-19, and
- can be used to find out if someone has already had COVID-19.
The immune system of people who have COVID-19 responds by developing proteins in the blood called antibodies that attack the virus. Detecting antibodies in people’s blood may indicate whether they currently have COVID-19 or have had it previously. Whilst detecting current infection is usually done using swab tests within the first 5 days of illness, they may miss infection and are not available to all.
So the question is: are COVID-19 antibody tests accurate enough to meet their intended purpose?
Cochrane, published a review of studies looking at the accuracy of covid-19 antibody tests. Their conclusions:
- The timing of the test is vital. Use them at the wrong time and they don’t work.
- We don’t really know how accurately they identify COVID-19 in people with mild or no symptoms, or tested more than five weeks after symptoms started.
- Studies were small and did not report their results fully.
- Many papers included multiple samples from the same patients.
- More than half of the studies were made available before they had been through peer review.
Recommendation from a Six Sigma expert
If we want to track COVID-19 accurately, we need to modify our approach to analyzing test results based on the Cochrane review findings. If this were a Six Sigma project, here’s what I would recommend to a Six Sigma Black Belt:
- Review all test reports, both current and past.
- Discard any test where the person tested was asymptomatic or who had only mild symptoms.
- Discard tests taken less than 15 days from the onset of symptoms.
- Discard tests taken 35 or more days from the onset of symptoms.
- Evaluate trends in addition to the absolute number of positive or negative test results. This would improve decision making by factoring out false positive and false negative results.
- The baseline for measuring trends would be the average number and rate of positive test results.
- Stratify the data by various classifications: age group, geographic location, pre-existing conditions, etc.. Statistically compare the different classification results and use significant differences to help guide decision making.
- Evaluate the severity of the disease. Were patients eventually cured? Hospitalized? Put into ICU? Put on a ventilator? Did they die?
Of course, as with any Six Sigma project, the Black Belt should not do the analysis on hers or his own. Assemble a team of healthcare professionals and epidemiologists to guide the collection and analysis of data. Present the results to decision makers interested in the big picture beyond health related decisions, such as economists, business leaders, politicians, and other stakeholders interested in the outcomes of any decision. The ultimate goal is to make decisions based on evidence and risk assessment using accurate data.
Take a tour! In this interactive video you will discover what you will learn in our Lean Six Sigma Black Belt training and certification program.