Observational studies and the difficult quest for causality: lessons from vaccine effectiveness and impact studies

Abstract

Although randomized placebo-controlled trials (RCT) are critical to establish efficacy of vaccines at the time of licensure, important remaining questions about vaccine effectiveness (VE)-used here to include individual-level measures and population-wide impact of vaccine programmes-can only be answered once the vaccine is in use, from observational studies. However, such studies are inherently at risk for bias. Using a causal framework and illustrating with examples, we review newer approaches to detecting and avoiding confounding and selection bias in three major classes of observational study design: cohort, case-control and ecological studies. Studies of influenza VE, especially in seniors, are an excellent demonstration of the challenges of detecting and reducing such bias, and so we use influenza VE as a running example. We take a fresh look at the time-trend studies often dismissed as 'ecological'. Such designs are the only observational study design that can measure the overall effect of a vaccination programme [indirect (herd) as well as direct effects], and are in fact already an important part of the evidence base for several vaccines currently in use. Despite the great strides towards more robust observational study designs, challenges lie ahead for evaluating best practices for achieving robust unbiased results from observational studies. This is critical for evaluation of national and global vaccine programme effectiveness.

Keywords: case-control studies; ecological study; epidemiologic methods; negative control; observational studies; test-negative design.

Figure 1.

Causal structure of observational VE studies showing the possible confounding of the vaccine (V)-outcome (e.g. hospitalization or death D) relationship by health status, health-seeking behaviour or other characteristics that may differ between those who do and do not receive vaccination. Any causal effect of vaccination on D must be mediated by I, infection with the pathogen against which the vaccine protects.

Figure 2.

As Figure 1, but showing a negative-control outcome D’ (for example death outside influenza season) that has no causal connection to vaccination but shares the same confounding relationship with H as the outcome of interest D.

Figure 3.

Causal diagrams of the test-negative design. Here, the outcome of interest is laboratory-confirmed clinically attended influenza infection (T+). A: As before, health-seeking behaviur or other characteristics H may differ between those who do and do not receive vaccination, and may affect the probability of receiving a test for influenza (T) through other confounding pathways (H- > T). The test-negative design conditions on T by including only those who are tested, and thereby blocks this confounding effect. B: However, this conditioning creates selection bias, a non-causal association (orange line 1) between health-seeking behaviour and infection, by conditioning on their common effect, biasing the V- > T+ association. C: The test-negative design in greater detail, including two time periods (e.g. consecutive weeks) in which individuals are enrolled. The study is intended to measure protection by the pathways shown in green. Bias may occur if (arrow 2a) vaccination has a short-term, non-specific effect on other infections N, or (arrow 2b) if influenza is temporarily protective against other infections N or (arrow 2c) if influenza is protective against influenza later in the season. D: The test-negative design does not protect (nor does the ‘classic’ case-control design) against confounding effects in which some factor G (e.g. being a health-care worker) is a common cause of both getting vaccinated and getting infected, given vaccination status.

Figure 4.

In a time-trend study of vaccine effects, year (Y) affects vaccine coverage C, which can be interpreted as each individual’s risk of being vaccinated V. Coverage also affects the vaccination status of contacts V’, which affects (through herd immunity) an individual’s infection risk. The association between coverage and the outcome D (typically a severe one, such as pneumonia or death) will be causal if year is affecting the outcome only through vaccine coverage and not through other paths such as development—that is, if the orange arrow is absent.