Distinguishing Selection Bias and Confounding Bias in Comparative Effectiveness Research

Abstract

Comparative effectiveness research (CER) aims to provide patients and physicians with evidence-based guidance on treatment decisions. As researchers conduct CER they face myriad challenges. Although inadequate control of confounding is the most-often cited source of potential bias, selection bias that arises when patients are differentially excluded from analyses is a distinct phenomenon with distinct consequences: confounding bias compromises internal validity, whereas selection bias compromises external validity. Despite this distinction, however, the label "treatment-selection bias" is being used in the CER literature to denote the phenomenon of confounding bias. Motivated by an ongoing study of treatment choice for depression on weight change over time, this paper formally distinguishes selection and confounding bias in CER. By formally distinguishing selection and confounding bias, this paper clarifies important scientific, design, and analysis issues relevant to ensuring validity. First is that the 2 types of biases may arise simultaneously in any given study; even if confounding bias is completely controlled, a study may nevertheless suffer from selection bias so that the results are not generalizable to the patient population of interest. Second is that the statistical methods used to mitigate the 2 biases are themselves distinct; methods developed to control one type of bias should not be expected to address the other. Finally, the control of selection and confounding bias will often require distinct covariate information. Consequently, as researchers plan future studies of comparative effectiveness, care must be taken to ensure that all data elements relevant to both confounding and selection bias are collected.

Figure 1

Interplay between the study population and study sub-sample in the context of distinguishing external from internal validity. Numbers in the parentheses correspond to the observed sample sizes in the motivating comparative effectiveness study of treatment for depression and two-year weight change.

Figure 2

Directed acyclic graphs illustrating the potential for confounding bias and selection bias under a randomized trial (RT) or an observational study (OS), with various scenarios for the selection mechanism. In each sub-figure, Rx is treatment choice, Y is the outcome of interest and L is a collection of factors related to the outcome and, possibly, treatment choice. The boxed “S=1” indicates that analyses are only performed on patients selected into the study sub-sample.

Figure 3

A simplified directed acyclic graph for the motivating comparative effectiveness study of treatment for depression (Rx) on two-year weight change (Y). Baseline smoking (L₁) and weight (L₂) are confounders of the association of interest; gender (L₃) is associated with weight change but independent of treatment choice. Baseline smoking and gender are determinants of selection into the study sub-sample but being associated with whether or not a patient has complete data in the EMR. Also shown are various models relevant to the adjustment of confounding bias and selection bias.