Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 May 1;8(5):e2512763.
doi: 10.1001/jamanetworkopen.2025.12763.

Evaluating the Test-Negative Design for COVID-19 Vaccine Effectiveness Using Randomized Trial Data: A Secondary Cross-Protocol Analysis of 5 Randomized Clinical Trials

Leah I B Andrews  1 , M Elizabeth Halloran  1   2 , Kathleen M Neuzil  3 , Lars van der Laan  4 , Yunda Huang  2 , Jessica Andriesen  2 , Mayur Patel  2 , Leigh H Fisher  2 , Holly Janes  2 , Nadine Rouphael  5 , Stephen R Walsh  6 , Deborah A Theodore  7 , Hong-Van Tieu  7   8 , Magdalena Sobieszczyk  7 , Hana M El Sahly  9 , Lindsey R Baden  6 , Ann R Falsey  10 , Thomas B Campbell  11 , Colleen F Kelley  5 , Catherine Mary Healy  12 , Lilly Immergluck  13 , Benjamin Luft  14 , Ian Hirsch  15 , Guy de Bruyn  16 , Carla Truyers  17 , Frances Priddy  18 , Kelsey M Sumner  19 , Brendan Flannery  19 , Dean Follmann  20 , Peter B Gilbert  1   2 ; COVID-19 Prevention Network (CoVPN)
Collaborators, Affiliations

Evaluating the Test-Negative Design for COVID-19 Vaccine Effectiveness Using Randomized Trial Data: A Secondary Cross-Protocol Analysis of 5 Randomized Clinical Trials

Leah I B Andrews et al. JAMA Netw Open. .

Abstract

Importance: The test-negative design (TND) has been widely used to assess postmarketing COVID-19 vaccine effectiveness but requires further evaluation for this application.

Objective: To determine whether the TND reliably evaluates vaccine effectiveness against symptomatic COVID-19 using placebo-controlled vaccine efficacy randomized clinical trials (RCTs).

Design, setting, and participants: This secondary cross-protocol analysis constructed TND study datasets from study sites in 16 countries across 5 continents using the blinded phase cohorts of 5 harmonized phase 3 COVID-19 Prevention Network RCTs: COVE (Coronavirus Vaccine Efficacy and Safety), AZD1222, ENSEMBLE, PREVENT-19 (Prefusion Protein Subunit Vaccine Efficacy Novavax Trial COVID-19), and VAT00008. Participants included adults who received the intended number of doses, experienced COVID-19-like symptoms, and obtained SARS-CoV-2 testing. Start dates ranged from July 27, 2020, to October 19, 2021; data cutoff dates ranged from March 26, 2021, to March 15, 2022. Statistical analysis was performed from May 11, 2023, to February 25, 2025.

Interventions: Participants received vaccines consisting of messenger RNA-1273 (COVE; 2 doses 28 days apart), ChAdOx1 nCoV-19 (AZD1222; 2 doses 28 days apart), Ad26.COV2.S (ENSEMBLE; 1 dose), NVX-CoV2373 (PREVENT-19; 2 doses 21 days apart), CoV2 preS dTM-AS03 (VAT00008; D614) (2 doses 21 days apart), or CoV2 preS dTM-AS03 (D614 plus B.1.351) (VAT00008; 2 doses 21 days apart) or placebo.

Main outcomes and measures: Main outcomes were symptomatic COVID-19 according to each trial's primary efficacy definition and the Centers for Disease Control and Prevention definition. Vaccine effectiveness was estimated using targeted maximum likelihood estimation under a semiparametric logistic regression model and ordinary logistic regression. Noncase exchangeability, a core TND assumption for unbiased estimation, was also assessed by estimating vaccine efficacy against non-COVID-19 illness.

Results: Among the 12 157 participants included in the analysis, mean (SD) age was 45 (15) years, 6414 were female (53%), 5858 were vaccinated (48%), 2835 experienced primary COVID-19 (23%), and 2992 experienced Centers for Disease Control and Prevention-defined COVID-19 (25%). TND vaccine effectiveness estimates were concordant with RCT vaccine efficacy estimates (concordance correlation coefficient, 0.86 [95% CI, 0.58-0.96] for both outcomes). The semiparametric method had 48% smaller variance estimates than ordinary logistic regression. Noncase exchangeability was generally supported with a median vaccine efficacy against non-COVID-19 illness of 7.7% (IQR, 2.7%-16.8%) across trial cohorts and most 95% CIs including 0.

Conclusions and relevance: In this cross-protocol analysis, the TND provided reliable inferences on COVID-19 vaccine effectiveness in health care-seeking populations for multiple vaccines and symptom definitions when confounding and selection bias were absent. A machine-learning approach for robust confounding control in postmarketing TND studies was also introduced.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Ms Andrews reported receiving grant support from National Institute of Allergy and Infectious Diseases (NIAID) during the conduct of the study and conference attendance and travel support from the University of Washington Department of Biostatistics and University of Washington Graduate School during the conduct of the study. Dr Neuzil reported receiving grant support from Pfizer Inc and the National Institutes of Health (NIH) during the conduct of the study. Dr Huang reported receiving grant support from the NIH during the conduct of the study. Dr Andriesen reported receiving grant support from US NIH during the conduct of the study. Dr Janes reported receiving grant support from the NIH during the conduct of the study. Dr Rouphael reported receiving grant support from the NIH and Sanofi SA during the conduct of the study and funding to institution from Sanofi SA, Eli Lilly and Company, Merck & Co Inc, QuidelOrtho Corporation, Immorna Biotherapeutics, and Pfizer Inc, serving on selected advisory boards for Sanofi SA, CSL Seqirus, Pfizer Inc, and Moderna Inc, and serving as a paid clinical trials safety consultant for ICON PLC, CyanVac LLC, Imunon Inc, and The Emmes Company LLC. Dr Walsh reported receiving grant support from the NIAID and Sanofi Pasteur and nonfinancial support from Sanofi Pasteur during the conduct of the study and grant support from Moderna Inc, Vir Biotechnology Inc, Worcester HIV Vaccine, Pfizer Inc, Janssen Global Services Inc, and AbbVie Inc, personal fees from Janssen Global Services Inc and BioNTech SE outside the submitted work, and having a spouse who is an employee of Regeneron Pharmaceuticals Inc and holds stock/stock options. Dr Tieu reported receiving grant support from New York Blood Center during the conduct of the study. Dr Sobieszczyk reported receiving grant support to institution from the NIH during the conduct of the study and grant support to institution from the Gates Foundation, Sanofi SA, Merck Sharpe and Dohme, and Gilead Sciences Inc outside the submitted work. Dr El Sahly reported receiving grant support from the NIAID during the conduct of the study. Dr Baden reported receiving grant support from the NIH during the conduct of the study. Dr Falsey reported receiving grant support from AstraZeneca during the conduct of the study and grant support from AstraZeneca, Pfizer Inc, and CyanVac LLC and personal fees for serving on an advisory board from Merck & Co Inc, GSK, ADMA Biologics Inc, and Moderna Inc outside the submitted work. Dr Kelley reported receiving grant support to institution from Moderna Inc and Novavax Inc during the conduct of the study and grant support to institution from Gilead Sciences Inc, ViiV Healthcare, and Humanigen Inc outside the submitted work. Dr Healy reported receiving grant support to institution from the NIH during the conduct of the study and ownership of stocks from QuidelOrtho Corporation outside the submitted work. Dr Immergluck reported receiving funding from Merck & Co Inc and GSK to support clinical trials in children and grants from Novavax Inc and Pfizer Inc to provide vaccine for trials outside the submitted work. Dr Hirsch reported holding stock or stock options in AstraZeneca. Dr de Bruyn reported holding shares or share options from Sanofi SA during the conduct of the study and having a patent pending for Sanofi Pasteur Development of CoV-2 dTM vaccine. Dr Gilbert reported receiving grant support to institution from the NIAID and NIH during the conduct of the study and serving on the vaccine scientific advisory boards for Moderna Inc and AstraZeneca. No other disclosures were reported.

Figures

Figure 1.
Figure 1.. Primary COVID-19 Vaccine Efficacy and Semiparametric Logistic Regression Vaccine Effectiveness Estimates by Sampling Method
Vaccine efficacy was estimated from 10 phase 3 COVID-19 Prevention Network randomized clinical trial (RCT) cohorts using the primary statistical approach in the final blinded phase analysis publications (Methods section and eTable 1 in Supplement 1).,,,,, Test-negative design (TND) study datasets were constructed from the RCT cohorts using 4 TND sampling methods (Methods section). Vaccine effectiveness was defined as 1 minus the COVID-19 conditional risk ratio (vaccine vs placebo) and estimated on the TND study datasets using targeted maximum likelihood estimation under a semiparametric logistic regression model that adjusts for age, sex, race and ethnicity, region, comorbidities, and testing date (Methods section and eTable 2 in Supplement 1). Estimates and 95% CIs are compared on the natural logarithm (1 minus vaccine efficacy or effectiveness) scale, with plotting labels on the vaccine efficacy or effectiveness scale. BN indicates baseline SARS-CoV-2 negative; BP, baseline SARS-CoV-2 positive; LA, Latin America; S1, stage 1; S2, stage 2; and SA, South Africa.
Figure 2.
Figure 2.. Placebo-Controlled Randomized Clinical Trial (RCT) Vaccine Efficacy Estimates vs Test-Negative Design (TND) Semiparametric Logistic Regression Vaccine Effectiveness Estimates
A, Primary COVID-19 vaccine efficacy estimates from RCT cohorts and semiparametric logistic regression vaccine effectiveness estimates from primary COVID-19 TND participant-based samples with censoring for COVID-19. B, Centers for Disease Control and Prevention (CDC) COVID-19 vaccine efficacy estimates from RCT cohorts and semiparametric logistic regression vaccine effectiveness estimates from CDC COVID-19 TND participant-based samples with censoring for COVID-19. C, Primary COVID-19 vaccine efficacy estimates from RCT cohorts and semiparametric logistic regression vaccine effectiveness estimates from primary COVID-19 TND random specimen-based samples. D, CDC COVID-19 vaccine efficacy estimates from RCT cohorts and semiparametric logistic regression vaccine effectiveness estimates from CDC COVID-19 TND random specimen-based samples. Vaccine efficacy and effectiveness were estimated from 10 final blinded phase, primary efficacy analysis cohorts from 5 phase 3 COVID-19 Prevention Network RCTs.,,,,, Vaccine efficacy was estimated using each trial’s primary efficacy analysis approach for primary COVID-19,,,,, and an unadjusted Cox proportional hazards model for CDC COVID-19. Vaccine effectiveness was estimated using targeted maximum likelihood estimation under a semiparametric logistic regression model that adjusted for age, sex, race and ethnicity, region, comorbidities, and testing date. Estimates (symbols) and 95% CIs (vertical and horizontal line segments) are compared on the natural logarithm (1 minus vaccine efficacy or effectiveness) scale, with plotting labels on the vaccine efficacy or effectiveness scale. The 95% CI lower bounds for VAT00008 Stage 2 baseline SARS-CoV-2 negative (BN) TND estimates extend beyond the plotting region. Concordance correlation coefficient (CCC) estimates and 95% CIs are reported. BP indicates baseline SARS-CoV-2 positive; COVE Coronavirus Vaccine Efficacy and Safety; LA, Latin America; PREVENT-19, Prefusion Protein Subunit Vaccine Efficacy Novavax Trial COVID-19; S1, stage 1; S2, stage 2; and SA, South Africa.
Figure 3.
Figure 3.. Vaccine Efficacy (VE) Against Non–COVID-19 Illness to Assess Noncase Exchangeability Violations
VE against non–COVID-19 illness was defined as 1 minus the hazard ratio (vaccine vs placebo) of non–COVID-19 illness, estimated from an unadjusted Cox proportional hazards model using the Efron method for handling ties and score test P values. Models were fit on the overall randomized clinical trial (RCT) cohort and on subgroups younger than 60 years and 60 years or older for each final blinded phase of the phase 3 RCT primary efficacy analysis cohorts: COVE (Coronavirus Vaccine Efficacy and Safety) baseline SARS-CoV-2 negative (BN), AZD1222 BN, ENSEMBLE BN (analyzed separately as Latin America [LA], South Africa [SA], and the US), PREVENT-19 (Prefusion Protein Subunit Vaccine Efficacy Novavax Trial COVID-19) BN, and VAT00008 (analyzed separately by stage 1 [S1] monovalent and stage 2 [S2] bivalent vaccine BN and BP). Estimates and 95% CIs are compared on the natural logarithm (1 minus VE) scale, with plotting labels on the VE scale. Vaccine efficacy was not estimated in subgroups with no non–COVID-19 illness cases. BP indicates baseline SARS-CoV-2 positive; NE, not estimated.
See this image and copyright information in PMC

References

    1. El Sahly HM, Baden LR, Essink B, et al. ; COVE Study Group . Efficacy of the mRNA-1273 SARS-CoV-2 vaccine at completion of blinded phase. N Engl J Med. 2021;385(19):1774-1785. doi:10.1056/NEJMoa2113017 - DOI - PMC - PubMed
    1. Sobieszczyk ME, Maaske J, Falsey AR, et al. ; AstraZeneca AZD1222 Clinical Study Group . Durability of protection and immunogenicity of AZD1222 (ChAdOx1 nCoV-19) COVID-19 vaccine over 6 months. J Clin Invest. 2022;132(18):e160565. doi:10.1172/JCI160565 - DOI - PMC - PubMed
    1. Sadoff J, Gray G, Vandebosch A, et al. ; ENSEMBLE Study Group . Final analysis of efficacy and safety of single-dose Ad26.COV2.S. N Engl J Med. 2022;386(9):847-860. doi:10.1056/NEJMoa2117608 - DOI - PMC - PubMed
    1. Dunkle LM, Kotloff KL, Gay CL, et al. ; 2019nCoV-301 Study Group . Efficacy and safety of NVX-CoV2373 in adults in the United States and Mexico. N Engl J Med. 2022;386(6):531-543. doi:10.1056/NEJMoa2116185 - DOI - PMC - PubMed
    1. Dayan GH, Rouphael N, Walsh SR, et al. ; VAT00008 study team . Efficacy of a monovalent (D614) SARS-CoV-2 recombinant protein vaccine with AS03 adjuvant in adults: a phase 3, multi-country study. EClinicalMedicine. 2023;64:102168. doi:10.1016/j.eclinm.2023.102168 - DOI - PMC - PubMed

Publication types

Substances