Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Jan 27:9:e58981.
doi: 10.2196/58981.

Methods to Adjust for Confounding in Test-Negative Design COVID-19 Effectiveness Studies: Simulation Study

Elizabeth Ak Rowley  1 , Patrick K Mitchell  1 , Duck-Hye Yang  1 , Ned Lewis  2 , Brian E Dixon  3   4 , Gabriela Vazquez-Benitez  5 , William F Fadel  3   4 , Inih J Essien  5 , Allison L Naleway  6 , Edward Stenehjem  7 , Toan C Ong  8 , Manjusha Gaglani  9   10 , Karthik Natarajan  11   12 , Peter Embi  3   13 , Ryan E Wiegand  14 , Ruth Link-Gelles  14 , Mark W Tenforde  14 , Bruce Fireman  2
Affiliations

Methods to Adjust for Confounding in Test-Negative Design COVID-19 Effectiveness Studies: Simulation Study

Elizabeth Ak Rowley et al. JMIR Form Res. .

Abstract

Background: Real-world COVID-19 vaccine effectiveness (VE) studies are investigating exposures of increasing complexity accounting for time since vaccination. These studies require methods that adjust for the confounding that arises when morbidities and demographics are associated with vaccination and the risk of outcome events. Methods based on propensity scores (PS) are well-suited to this when the exposure is dichotomous, but present challenges when the exposure is multinomial.

Objective: This simulation study aimed to investigate alternative methods to adjust for confounding in VE studies that have a test-negative design.

Methods: Adjustment for a disease risk score (DRS) is compared with multivariable logistic regression. Both stratification on the DRS and direct covariate adjustment of the DRS are examined. Multivariable logistic regression with all the covariates and with a limited subset of key covariates is considered. The performance of VE estimators is evaluated across a multinomial vaccination exposure in simulated datasets.

Results: Bias in VE estimates from multivariable models ranged from -5.3% to 6.1% across 4 levels of vaccination. Standard errors of VE estimates were unbiased, and 95% coverage probabilities were attained in most scenarios. The lowest coverage in the multivariable scenarios was 93.7% (95% CI 92.2%-95.2%) and occurred in the multivariable model with key covariates, while the highest coverage in the multivariable scenarios was 95.3% (95% CI 94.0%-96.6%) and occurred in the multivariable model with all covariates. Bias in VE estimates from DRS-adjusted models was low, ranging from -2.2% to 4.2%. However, the DRS-adjusted models underestimated the standard errors of VE estimates, with coverage sometimes below the 95% level. The lowest coverage in the DRS scenarios was 87.8% (95% CI 85.8%-89.8%) and occurred in the direct adjustment for the DRS model. The highest coverage in the DRS scenarios was 94.8% (95% CI 93.4%-96.2%) and occurred in the model that stratified on DRS. Although variation in the performance of VE estimates occurred across modeling strategies, variation in performance was also present across exposure groups.

Conclusions: Overall, models using a DRS to adjust for confounding performed adequately but not as well as the multivariable models that adjusted for covariates individually.

Keywords: COVID-19; assessment; comorbidity; disease risk score; propensity score; simulation study; usefulness; vaccine effectiveness.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: None declared.

Figures

Figure 1
Figure 1
Summary of simulation results in the overall sample of 10,000. DRS: disease risk score; VE: vaccine effectiveness.
Figure 2
Figure 2
Summary of simulation results in the immunocompromised subset of 10,000. DRS: disease risk score; IC: immunocompromised; VE: vaccine effectiveness.
Figure 3
Figure 3
Summary of simulation results in the 50 years or older subset of 10,000. DRS: disease risk score; VE: vaccine effectiveness.
Figure 4
Figure 4
Summary of simulation results in a sample of size 1000. DRS: disease risk score; VE: vaccine effectiveness.
See this image and copyright information in PMC

References

    1. Arbogast PG, Kaltenbach L, Ding H, Ray WA. Adjustment for multiple cardiovascular risk factors using a summary risk score. Epidemiology. 2008;19(1):30–37. doi: 10.1097/EDE.0b013e31815be000. - DOI - PubMed
    1. Arbogast P, Seeger J, DMCSVW Group . Effective health care program research report. Agency for Healthcare Research and Quality; 2012. [2024-12-19]. Summary variables in observational research: propensity scores and disease risk scores. https://effectivehealthcare.ahrq.gov/products/observational-research-sco... .
    1. Arbogast PG, Ray WA. Use of disease risk scores in pharmacoepidemiologic studies. Stat Methods Med Res. 2009;18(1):67–80. doi: 10.1177/0962280208092347.0962280208092347 - DOI - PubMed
    1. Arbogast PG, Ray WA. Performance of disease risk scores, propensity scores, and traditional multivariable outcome regression in the presence of multiple confounders. Am J Epidemiol. 2011;174(5):613–620. doi: 10.1093/aje/kwr143.kwr143 - DOI - PubMed
    1. Miettinen OS. Stratification by a multivariate confounder score. Am J Epidemiol. 1976;104(6):609–620. doi: 10.1093/oxfordjournals.aje.a112339. - DOI - PubMed

Substances

LinkOut - more resources