Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages

Abstract

Evidence-based medicine is the application of scientific evidence to clinical practice. This article discusses the difficulties of applying global evidence ("average effects" measured as population means) to local problems (individual patients or groups who might depart from the population average). It argues that the benefit or harm of most treatments in clinical trials can be misleading and fail to reveal the potentially complex mixture of substantial benefits for some, little benefit for many, and harm for a few. Heterogeneity of treatment effects reflects patient diversity in risk of disease, responsiveness to treatment, vulnerability to adverse effects, and utility for different outcomes. Recognizing these factors, researchers can design studies that better characterize who will benefit from medical treatments, and clinicians and policymakers can make better use of the results.

Figure 1. Distribution of Individual-Specific Treatment Effects (ITEs) in Three Hypothetical Populations

The average treatment effect (ATE) is indicated by the vertical bar. Individuals to the right of the bar derive a greater than average benefit, while those to the left derive less than an average benefit or even harm. The horizontal axis is standardized by the pooled standard deviation of the outcome. The gray zone represents patients in whom the effect size is so small (+/−0.25 SD) as to be clinically meaningless. Panel A represents a normal distribution centered on an effect size of 0.5 SD; panel B represents a more narrow distribution, still centered on the same mean; and panel C is normally distributed but shifted to the right (mean effect size about 1.3 SDs).

Figure 2. Distribution of ITE in the Population (Large Unshaded Curve) and in Three Hypothetical Samples (Shaded Curves)

Sample 1 is centered but fails to reflect the diversity of the population in terms of net treatment benefit. Sample 2 is composed of individuals who happen to derive much more net benefit from the treatment than does the average member of the population. Only sample 3 is broadly representative of the population in terms of risk, responsiveness, and vulnerability.

Figure 3

Relationships of Risk, Responsiveness, and Vulnerability to Treatment Thresholds