Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Observational Study
. 2023 May;176(5):685-693.
doi: 10.7326/M21-4269. Epub 2023 May 2.

Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data

William J Hulme  1 , Elizabeth Williamson  2 , Elsie M F Horne  3 , Amelia Green  1 , Helen I McDonald  2 , Alex J Walker  1 , Helen J Curtis  1 , Caroline E Morton  1 , Brian MacKenna  1 , Richard Croker  1 , Amir Mehrkar  1 , Seb Bacon  1 , David Evans  1 , Peter Inglesby  1 , Simon Davy  1 , Krishnan Bhaskaran  2 , Anna Schultze  2 , Christopher T Rentsch  2 , Laurie Tomlinson  2 , Ian J Douglas  2 , Stephen J W Evans  2 , Liam Smeeth  2 , Tom Palmer  3 , Ben Goldacre  1 , Miguel A Hernán  4 , Jonathan A C Sterne  5
Affiliations
Observational Study

Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data

William J Hulme et al. Ann Intern Med. 2023 May.

Erratum in

Abstract

The COVID-19 vaccines were developed and rigorously evaluated in randomized trials during 2020. However, important questions, such as the magnitude and duration of protection, their effectiveness against new virus variants, and the effectiveness of booster vaccination, could not be answered by randomized trials and have therefore been addressed in observational studies. Analyses of observational data can be biased because of confounding and because of inadequate design that does not consider the evolution of the pandemic over time and the rapid uptake of vaccination. Emulating a hypothetical "target trial" using observational data assembled during vaccine rollouts can help manage such potential sources of bias. This article describes 2 approaches to target trial emulation. In the sequential approach, on each day, eligible persons who have not yet been vaccinated are matched to a vaccinated person. The single-trial approach sets a single baseline at the start of the rollout and considers vaccination as a time-varying variable. The nature of the confounding depends on the analysis strategy: Estimating "per-protocol" effects (accounting for vaccination of initially unvaccinated persons after baseline) may require adjustment for both baseline and "time-varying" confounders. These issues are illustrated by using observational data from 2 780 931 persons in the United Kingdom aged 70 years or older to estimate the effect of a first dose of a COVID-19 vaccine. Addressing the issues discussed in this article should help authors of observational studies provide robust evidence to guide clinical and policy decisions.

PubMed Disclaimer

Conflict of interest statement

Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M21-4269.

Figures

Figure 1.
Figure 1.. Coverage of first and second dose of BNT162b2 and ChAdOx1 vaccination.
Coverage on each day was calculated as 10 000 times the number of persons in each status, divided by the number of persons alive and registered. BNT162b2 = Pfizer–BioNTech BNT162b2 mRNA vaccine; ChAdOx1 = Oxford–AstraZeneca ChAdOx1 nCoV-19 AZD1222 vaccine.
Figure 2.
Figure 2.. Estimated vaccine effectiveness after ≥1 dose of BNT162b2 or ChAdOx1.
BNT162b2 = Pfizer–BioNTech BNT162b2 mRNA vaccine; ChAdOx1 = Oxford–AstraZeneca ChAdOx1 nCoV-19 AZD1222 vaccine.
See this image and copyright information in PMC

References

    1. Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol. 2016;183:758-764. [PMID: ] doi:10.1093/aje/kwv254 - DOI - PMC - PubMed
    1. UK Health Security Agency. COVID-19: the green book, chapter 14a. 27 November 2020. Accessed at www.gov.uk/government/publications/covid-19-the-green-book-chapter-14a on 13 May 2021.
    1. Hall VJ, Foulkes S, Saei A, et al; SIREN Study Group. COVID-19 vaccine coverage in health-care workers in England and effectiveness of BNT162b2 mRNA vaccine against infection (SIREN): a prospective, multicentre, cohort study. Lancet. 2021;397:1725-1735. [PMID: ] doi:10.1016/S0140-6736(21)00790-X - DOI - PMC - PubMed
    1. Lopez Bernal J, Andrews N, Gower C, et al. Effectiveness of the Pfizer–BioNTech and Oxford–AstraZeneca vaccines on Covid-19 related symptoms, hospital admissions, and mortality in older adults in England: test negative case-control study. BMJ. 2021;373:n1088. [PMID: ] doi:10.1136/bmj.n1088 - DOI - PMC - PubMed
    1. Vasileiou E, Simpson CR, Shi T, et al. Interim findings from first-dose mass COVID-19 vaccination roll-out and COVID-19 hospital admissions in Scotland: a national prospective cohort study. Lancet. 2021;397:1646-1657. [PMID: ] doi:10.1016/S0140-6736(21)00677-2 - DOI - PMC - PubMed

Publication types