How to do (or not to do) ... Assessing the impact of a policy change with routine longitudinal data

Abstract

A lack of good quality evidence on the effect of alternative social policies in low- and middle-income countries has been recently underlined and the value of randomized trials increasingly advocated. However, it is also acknowledged that randomization is not always feasible or politically acceptable. Analyses using longitudinal data series before and after an intervention can also deliver robust results and such data are often reasonably easy to access. Using the example of evaluating the impact on utilization of a change in health financing policy, this article explains how studies in the literature have often failed to address the possible biases that can arise in a simple analysis of routine longitudinal data. It then describes two possible statistical approaches to estimate impact in a more reliable manner and illustrates in detail the more simple method. Advantages and limitations of this quasi-experimental approach to evaluating the impact of health policies are discussed.