Simple graphical rules for assessing selection bias in general-population and selected-sample treatment effects

Abstract

When analyzing a selected sample from a general population, selection bias can arise relative to the causal average treatment effect (ATE) for the general population, and also relative to the ATE for the selected sample itself. In this paper, we provide simple graphical rules that indicate (1) whether a selected-sample analysis will be unbiased for each ATE and (2) whether adjusting for certain covariates could eliminate selection bias. The rules can easily be checked in a standard single-world intervention graph. When the treatment could affect selection, a third estimand of potential scientific interest is the "net treatment difference"-namely the net change in outcomes that would occur for the selected sample if all members of the general population were treated versus not treated, including any effects of the treatment on which individuals are in the selected sample. We provide graphical rules for this estimand as well. We decompose bias in a selected-sample analysis relative to the general-population ATE into (1) "internal bias" relative to the net treatment difference and (2) "net-external bias," a discrepancy between the net treatment difference and the general-population ATE. Each bias can be assessed unambiguously via a distinct graphical rule, providing new conceptual insight into the mechanisms by which certain causal structures produce selection bias.

Keywords: causal inference; collider stratification; effect-measure modification; generalizability; missing data; single-world intervention graphs.

Figure 1

Directed acyclic graphs representing various causal structures that may produce selection bias. A, treatment; Y, outcome; R, indicator for selection into analysis; V and W, additional variables that may or may not be measured among the selected sample. Panel (A): R is affected by A (via Y) but is not a collider. Panels (B)-(C): R is a collider.

Figure 2

For the example of a workplace program designed to reduce sick days, a depiction of how the selected sample (current employees; left) could change if the program were implemented universally (right). The 8 people in each panel depict the general population (all individuals who have ever been employed by the company); the solid person icons indicate current employees, and the unfilled person icons indicate individuals who are no longer employees. Yellow boxes indicate individuals who are enrolled in the program in each world; in the real world, only a subset of the general population is treated. Red heart symbols indicate individuals with preexisting health conditions; one such individual is not an employee in the real world, but would be an employee if the program were universally in effect.

Figure 3

Single-world intervention templates, , depicting a joint intervention on A and R.