The potential bias introduced into COVID-19 vaccine effectiveness studies at primary care level due to the availability of SARS-CoV-2 tests in the general population

Abstract

Background: With SARS-CoV-2 self-tests, persons with acute respiratory infections (ARI) can know their COVID-19 status. This may alter their decision to consult a general practitioner (GP), potentially biasing COVID-19 vaccine effectiveness (VE) studies. We explore bias mechanisms, simulate magnitude, and verify control methods.

Methods: We used directed acyclic graphs (DAGs) to illustrate the bias mechanisms. Based on the European primary care VEBIS multicentre test-negative design (TND) study, we simulated populations with varying true VE (20%-60%), proportions of persons with ARI self-testing (10%-30%), effect of COVID-19 vaccination on self-testing (1.5-2.5), and effect of self-test result on GP consultation (0.5-2). We performed 5000 runs per scenario, estimating VE among those consulting a GP. We calculated bias as true VE minus mean simulated VE, unadjusted and adjusted for self-testing, using logistic regression.

Results: DAGs suggested collider stratification bias if vaccination had an effect on self-testing and if self-test results affected GP consultation. Bias was -12% to 18% at 20% true VE, with the most extreme associations and 30% self-testing. With 60% true VE and 10%-20% self-testing, bias was lower. Bias was higher (-18% to 45%) if both positive and negative self-test results affected GP consultation. Adjusting for self-testing removed the bias.

Conclusions: Self-testing may bias COVID-19 VE TND studies in primary care if self-testing is high, particularly with low VE. We recommend primary care TND VE studies collect self-testing information to eliminate potential bias. Observational studies are needed to understand the relationship between vaccination, self-testing, and GP consultation, in these studies' source population.

Keywords: COVID-19; SARS-CoV-2; bias; primary health care; self-testing; vaccine effectiveness.

Figure 1.

Directed acyclic graph representing hypothesized causal relationships between key variables in a test-negative design study of COVID-19 vaccine effectiveness at primary care level, in the absence of self-test use. The effect of interest is that of COVID-19 vaccination on SARS-CoV-2 infection.

Figure 2.

Directed acyclic graph representing hypothesized causal relationships between variables in a test-negative design study of COVID-19 vaccine effectiveness at primary care level, in the context of self-test use, (A) without adjustment for self-testing and (B) with adjustment for self-testing. The effect of interest is that of COVID-19 vaccination on SARS-CoV-2 infection.

Figure 3.

Difference between the estimated VE and the true VE (mean bias), in a simulated primary care-based test-negative design study, under different scenarios defined by key parameters, in the context of self-test use, without adjustment for self-testing. The first set of plots presents results of simulations varying the probability of GP consultation after a positive self-test result (expressed relatively to that among those not self-testing). The second set of plots presents results of simulations varying the probability of GP consultation after a negative self-test result (expressed relatively to that among those not self-testing). The probability of self-testing among the vaccinated is expressed relatively to that among the unvaccinated. 10% self-testing in the unvaccinated population equates to 17% overall, 20% self-testing to 34% overall, and 30% self-testing to 50% overall, under the assumption of a 2.5 probability ratio for self-testing among the vaccinated population. P, probability; Vacc, vaccinated.

Figure 4.

Difference between the estimated VE and the true VE (mean bias), in a simulated primary care-based test-negative design study, with a probability ratio of self-testing of 2.5 among the vaccinated, 30% self-testing in the unvaccinated, and no effect of positive result on GP consultation, in the context of self-test use, with adjustment for self-testing. The probability of consultation after a negative self-test result is expressed relatively to that among those not self-testing.