How subgroup analyses can miss the trees for the forest plots: A simulation study

Abstract

Objectives: Subgroup analyses of clinical trial data can be an important tool for understanding when treatment effects differ across populations. That said, even effect estimates from prespecified subgroups in well-conducted trials may not apply to corresponding subgroups in the source population. While this divergence may simply reflect statistical imprecision, there has been less discussion of systematic or structural sources of misleading subgroup estimates.

Study design and setting: We use directed acyclic graphs to show how selection bias caused by associations between effect measure modifiers and trial selection, whether explicit (e.g., eligibility criteria) or implicit (e.g., self-selection based on race), can result in subgroup estimates that do not correspond to subgroup effects in the source population. To demonstrate this point, we provide a hypothetical example illustrating the sorts of erroneous conclusions that can result, as well as their potential consequences. We also provide a tool for readers to explore additional cases.

Conclusion: Treating subgroups within a trial essentially as random samples of the corresponding subgroups in the wider population can be misleading, even when analyses are conducted rigorously and all findings are internally valid. Researchers should carefully examine associations between (and consider adjusting for) variables when attempting to identify heterogeneous treatment effects.

Keywords: Subgroups; causal graphs; external validity; selection bias.

Figure 1

Two directed acyclic graphs. Panel A shows the causal relationships between V1, V2, X, participation, and Y in the source population; since there is no open path from V1 to Y in panel A, V1 will not modify the treatment effect of X on Y. Panel B shows the causal relationships between the same variables in the study population. Because we are conditioning on study participation, which depends on V1 and V2, there is now an open path from V1 to Y that can make V1 act as an effect modifier in the study population.

Figure 2

Directed acyclic graphs for when both V1 and V2 directly modify the treatment effect of X on Y. Panel A is the directed acyclic graph for the full population, while Panel B is the directed acyclic graph for the trial population. Both variables now have arrows indicating a causal effect on Y, and as a result both V1 and V2 are expected to have a different association with Y before and after conditioning on study participation.