Assessing regression to the mean effects in health care initiatives

Abstract

Background: Interventions targeting individuals classified as "high-risk" have become common-place in health care. High-risk may represent outlier values on utilization, cost, or clinical measures. Typically, such individuals are invited to participate in an intervention intended to reduce their level of risk, and after a period of time, a follow-up measurement is taken. However, individuals initially identified by their outlier values will likely have lower values on re-measurement in the absence of an intervention. This statistical phenomenon is known as "regression to the mean" (RTM) and often leads to an inaccurate conclusion that the intervention caused the effect. Concerns about RTM are rarely raised in connection with most health care interventions, and it is uncommon to find evaluators who estimate its effect. This may be due to lack of awareness, cognitive biases that may cause people to systematically misinterpret RTM effects by creating (erroneous) explanations to account for it, or by design.

Methods: In this paper, the author fully describes the RTM phenomenon, and tests the accuracy of the traditional approach in calculating RTM assuming normality, using normally distributed data from a Monte Carlo simulation and skewed data from a control group in a pre-post evaluation of a health intervention. Confidence intervals are generated around the traditional RTM calculation to provide more insight into the potential magnitude of the bias introduced by RTM. Finally, suggestions are offered for designing interventions and evaluations to mitigate the effects of RTM.

Results: On multivariate normal data, the calculated RTM estimates are identical to true estimates. As expected, when using skewed data the calculated method underestimated the true RTM effect. Confidence intervals provide helpful guidance on the magnitude of the RTM effect.

Conclusion: Decision-makers should always consider RTM to be a viable explanation of the observed change in an outcome in a pre-post study, and evaluators of health care initiatives should always take the appropriate steps to estimate the magnitude of the effect and control for it when possible. Regardless of the cause, failure to address RTM may result in wasteful pursuit of ineffective interventions, both at the organizational level and at the policy level.

Figure 1

Actual data illustrating the regression to the mean phenomenon in Coronary Artery Disease (CAD), Congestive Heart Failure (CHF), and Chronic Obstructive Pulmonary Disease (COPD). Quintile I is the lowest cost group and V the highest. All individuals were continuously enrolled during the 2-year period. The diagonal line represents perfect correlation between the first and second year costs, which can only be achieved in the complete absence of variability between measurements and no measurement error.

Figure 2

Physical Component Summary (PCS) scores on the Short Form-12 (SF-12v2), from a control group (n = 118) participating in a health coaching study (Butterworth et al. 2006). All participants were surveyed twice, once at program commencement and then again at three months. Squares/circles represent mean scores and capped lines represent 95% bootstrapped confidence intervals (1000 resamples).