Making the most of natural experiments: What can studies of the withdrawal of public health interventions offer?

Abstract

Many interventions that may have large impacts on health and health inequalities, such as social and public health policies and health system reforms, are not amenable to evaluation using randomised controlled trials. The United Kingdom Medical Research Council's guidance on the evaluation of natural experiments draws attention to the need for ingenuity to identify interventions which can be robustly studied as they occur, and without experimental manipulation. Studies of intervention withdrawal may usefully widen the range of interventions that can be evaluated, allowing some interventions and policies, such as those that have developed piecemeal over a long period, to be evaluated for the first time. In particular, sudden removal may allow a more robust assessment of an intervention's long-term impact by minimising 'learning effects'. Interpreting changes that follow withdrawal as evidence of the impact of an intervention assumes that the effect is reversible and this assumption must be carefully justified. Otherwise, withdrawal-based studies suffer similar threats to validity as intervention studies. These threats should be addressed using recognised approaches, including appropriate choice of comparators, detailed understanding of the change processes at work, careful specification of research questions, and the use of falsification tests and other methods for strengthening causal attribution. Evaluating intervention withdrawal provides opportunities to answer important questions about effectiveness of population health interventions, and to study the social determinants of health. Researchers, policymakers and practitioners should be alert to the opportunities provided by the withdrawal of interventions, but also aware of the pitfalls.

Keywords: Natural experiments; Policy evaluation; Research methods; Study design.

Fig. 1. An illustration of how causal effects may differ between evaluations of intervention introduction and withdrawal.

Scenario a: Evaluating the intervention's introduction provides a causal estimate that is more generalisable due to larger population coverage than studying partial withdrawal (A vs B). However, evaluating withdrawal provides a causal estimate that may be less prone to confounding than gradual introduction, since there is less chance of a large change in confounders over a shorter time period (D vs C). Scenario b: Evaluating the intervention's introduction estimates the causal effect before the intervention is optimised, whereas studying withdrawal allows the optimised causal effect to be estimated (E vs F).