A comparison of alternative strategies for choosing control populations in observational studies

Abstract

Various approaches have been used to select control groups in observational studies: (1) from within the intervention area; (2) from a convenience sample, or randomly chosen areas; (3) from areas matched on area-level characteristics; and (4) nationally. The consequences of the decision are rarely assessed but, as we show, it can have complex impacts on confounding at both the area and individual levels. We began by reanalyzing data collected for an evaluation of a rapid response service on rates of unplanned hospital admission. Balance on observed individual-level variables was better with external than local controls, after matching. Further, when important prognostic variables were omitted from the matching algorithm, imbalances on those variables were also minimized using external controls. Treatment effects varied markedly depending on the choice of control area, but in the case study the variation was minimal after adjusting for the characteristics of areas. We used simulations to assess relative bias and means-squared error, as this could not be done in the case study. A particular feature of the simulations was unexplained variation in the outcome between areas. We found that the likely impact of unexplained variation for hospital admissions dwarfed the benefits of better balance on individual-level variables, leading us to prefer local controls in this instance. In other scenarios, in which there was less unexplained variation in the outcome between areas, bias and mean-squared error were optimized using external controls. We identify some general considerations relevant to the choice of control population in observational studies.

Keywords: Program evaluation; Propensity score matching; Quasi-experiments.

Fig. 1

Area-level variables. The first panel shows the values of each of the 43 area-level variables under each strategy, while the second panel shows relative differences from the intervention area. Figures for Strategy 2 (‘random area’) are medians over all 32 possible geographies. Figures for Strategy 4 (‘national’) are weighted means over the 32 geographies (weighted for population size)

Fig. 2

Box plots of the estimated treatment effects based on 20,000 replications from the simulation experiment. The horizontal red line represents the true treatment effect. Saturation = 30 % and ${\mathit{\alpha }}_{2,1}=0.2$ throughout (Color figure online)