Effects of Confounding Bias in Coronavirus Disease 2019 (COVID-19) and Influenza Vaccine Effectiveness Test-Negative Designs Due to Correlated Influenza and COVID-19 Vaccination Behaviors

Abstract

Background: The test-negative design is commonly used to estimate influenza and coronavirus disease 2019 (COVID-19) vaccine effectiveness (VE). In these studies, correlated COVID-19 and influenza vaccine behaviors may introduce a confounding bias where controls are included with the other vaccine-preventable acute respiratory illness (ARI). We quantified the impact of this bias on VE estimates in studies where this bias is not addressed.

Methods: We simulated study populations under varying vaccination probabilities, COVID-19 VE, influenza VE, and proportions of controls included with the other vaccine-preventable ARI. Mean bias was calculated as the difference between estimated and true VE. Absolute mean bias in VE estimates was classified as low (<10%), moderate (10% to <20%), and high (≥20%).

Results: Where vaccination probabilities are positively correlated, COVID-19 and influenza VE test-negative studies with influenza and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) ARI controls, respectively, underestimate VE. For COVID-19 VE studies, mean bias was low for all scenarios where influenza represented ≤25% of controls. For influenza VE studies, mean bias was low for all scenarios where SARS-CoV-2 represented ≤10% of controls. Although bias was driven by the conditional probability of vaccination, low VE of the vaccine of interest and high VE of the confounding vaccine increase its magnitude.

Conclusions: Where a low percentage of controls is included with the other vaccine-preventable ARI, bias in COVID-19 and influenza VE estimates is low. However, influenza VE estimates are likely more susceptible to bias. Researchers should consider potential bias and its implications in their respective study settings to make informed methodological decisions in test-negative VE studies.

Keywords: COVID-19; SARS-CoV-2; influenza; negative; test; vaccine effectiveness.

Figure 1.

A, Simplified directed acyclic graph illustrating the relationship between coronavirus disease 2019 (COVID-19) and influenza vaccination probabilities. Vaccination motivation $M$is a common ancestor of influenza vaccine uptake $V_{flu}$ and COVID-19 vaccine uptake $V_{covid}$. The parentheses indicate that $M$is unmeasured. Through $M$a forked, confounding pathway exists linking $V_{covid}$ to medically attended influenza acute respiratory illness (ARI) $I_{flu}$ ($V_{covid}\leftarrow (M)\to I_{flu})$, 
and $V_{flu}$ to medically attended severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) ARI $I_{SARS-CoV-2}$ ($V_{flu}\leftarrow (M)\to I_{SARS-CoV-2})$. B, Adjustment in the statistical model for $V_{flu}$ closes the confounding pathway from $V_{covid}\leftarrow (M)\to I_{flu}$ in COVID-19 vaccine effectiveness (VE) test-negative studies that include influenza controls. C, Similarly, adjustment for $V_{covid}$ in an influenza VE test-negative study that includes SARS-CoV-2 controls closes the confounding pathway from $V_{flu}\leftarrow (M)\to I_{SARS-CoV-2}$.

Figure 2.

Exploration of bias in coronavirus disease 2019 (COVID-19) test-negative vaccine effectiveness (VE) studies that include influenza controls (A) and influenza test-negative VE studies that include severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) controls (B). Under the 2 assumptions that (i) influenza and COVID-19 vaccinations are protective against their respective diseases, and (ii) influenza and COVID-19 vaccination behaviors are positively correlated, inclusion of controls with the other vaccine-preventable acute respiratory illness will overrepresent unvaccinated controls ($d$). Overrepresentation of ($d$) increases the odds ratio (OR) comparing vaccination odds between cases and controls, and underestimates true VE given the formula VE = (1 – OR) × 100.

Figure 3.

Mean bias and 95% confidence intervals in coronavirus disease 2019 vaccine effectiveness estimates derived from a test-negative study with influenza controls under varying scenarios of $R{R}_{covidvx|fluvx}$, $V{E}_{flu}$, $V{E}_{covid}$, and $P_{controls}$. Abbreviations: COVID-19, coronavirus disease 2019; RR, risk ratio; VE, vaccine effectiveness.

Figure 4.

Mean bias and 95% confidence intervals in influenza vaccine effectiveness estimates derived from a test-negative study with severe acute respiratory syndrome coronavirus 2 controls under varying scenarios of $R{R}_{fluvx|covidvx}$, $V{E}_{flu}$, $V{E}_{covid}$, and $P_{controls}$. Abbreviations: COVID-19, coronavirus disease 2019; RR, risk ratio; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; VE, vaccine effectiveness.