Quasi-experimental study designs series-paper 6: risk of bias assessment

Abstract

Objectives: Rigorous and transparent bias assessment is a core component of high-quality systematic reviews. We assess modifications to existing risk of bias approaches to incorporate rigorous quasi-experimental approaches with selection on unobservables. These are nonrandomized studies using design-based approaches to control for unobservable sources of confounding such as difference studies, instrumental variables, interrupted time series, natural experiments, and regression-discontinuity designs.

Study design and setting: We review existing risk of bias tools. Drawing on these tools, we present domains of bias and suggest directions for evaluation questions.

Results: The review suggests that existing risk of bias tools provide, to different degrees, incomplete transparent criteria to assess the validity of these designs. The paper then presents an approach to evaluating the internal validity of quasi-experiments with selection on unobservables.

Conclusion: We conclude that tools for nonrandomized studies of interventions need to be further developed to incorporate evaluation questions for quasi-experiments with selection on unobservables.

Keywords: Difference in differences; Instrumental variables; Interrupted time series; Meta-Analysis; Natural experiment; Quasi-experiment; Regression discontinuity; Risk of bias; Systematic review.