Estimating Waning of Vaccine Effectiveness: A Simulation Study

Abstract

Background: Developing accurate and reliable methods to estimate vaccine protection is a key goal in immunology and public health. While several statistical methods have been proposed, their potential inaccuracy in capturing fast intraseasonal waning of vaccine-induced protection needs to be rigorously investigated.

Methods: To compare statistical methods for estimating vaccine effectiveness (VE), we generated simulated data using a multiscale, agent-based model of an epidemic with an acute viral infection and differing extents of VE waning. We apply a previously proposed framework for VE measures based on the observational data richness to assess changes of vaccine-induced protection over time.

Results: While VE measures based on hard-to-collect information (eg, the exact timing of exposures) were accurate, usually VE studies rely on time-to-infection data and the Cox proportional hazards model. We found that its extension using scaled Schoenfeld residuals, previously proposed for capturing VE waning, was unreliable in capturing both the degree of waning and its functional form and identified the mathematical factors contributing to this unreliability. We showed that partitioning time and including a time-vaccine interaction term in the Cox model significantly improved estimation of VE waning, even in the case of dramatic, rapid waning. We also proposed how to optimize the partitioning scheme.

Conclusions: While appropriate for rejecting the null hypothesis of no waning, scaled Schoenfeld residuals are unreliable for estimating the degree of waning. We propose a Cox-model-based method with a time-vaccine interaction term and further optimization of partitioning time. These findings may guide future analysis of VE waning data.

Keywords: Cox proportional hazards model; estimating waning of vaccine effectiveness; multiscale modeling; scaled Schoenfeld residuals; vaccine efficacy.

Figure 1.

Comparing the true vaccine effectiveness (VE) values at the individual and population levels for 4 scenarios. A, Four scenarios, where either the vaccine protection remains constant at 80% or protection wanes from 100% to 0% over 60 days, with or without spread of vaccination. B, Individual level of protection. C, The population average equals the individual level of protection if vaccination occurs on a single day. D, Average protection in the susceptible vaccinated population when vaccination is spread over 30 days. (See Equation SM 1 in the Supplementary Materials).

Figure 2.

Estimating vaccine effectiveness (VE) for constant (A, B, E, F) and waning (C, D, G, H) protection. When vaccine protection is constant at 80% for 1-day and 30-day spread of vaccination (A and B, respectively), all levels are reasonably accurate when infection numbers are high, as can be seen in their corresponding epidemic dynamics in E and F. If vaccination is spread, early results should be considered with caution owing to the extremely small relative size of the vaccinated group, which can lead to lack of infections and exposures in that group. C, D, When vaccine protection is waning, complications arise especially for level 3, which underestimates early season behavior. This is especially clear when vaccination is spread, as in D, with the level 3 scaled Schoenfeld residuals method (SR method) estimating accurately only at the very peak of infection. In A–D, light gray boxes display the region where total daily infections are low (<60 infections as suggested in [23]), and the x-axis ends on the day of the last infection.

Figure 3.

Inaccuracy not due to insufficient information for level 3. While the scaled Schoenfeld residuals method (SR method) for calculating level 3 values may fail in certain circumstances, level 3 estimates of vaccine effectiveness (VE) are not inherently inaccurate. With use of a time-vaccine interaction method (TVI method), as defined in the section Mathematical Factors Contributing to Inaccuracy of the SR Method, estimates for level 3 are quite similar to level 1 and 2 estimates.

Figure 4.

Approximations of level 3 methods can lead to substantial error in estimates of vaccine effectiveness (VE). With each approximation from the time-vaccine interaction (TVI) method, error is compounded. With the first approximation coming from the Taylor series (TS) and its weighted version, the Schoenfeld residuals method with time-dependent variance (SRTV), error increases, especially when the number of events is low. This error increases further under the additional assumption that variance is fixed, giving the standard SR method. This error is obvious qualitatively and also when quantified by the root mean square error (RMSE).

Figure 5.

Rather than using simple 100-event bins, the time-vaccine interaction method can be further improved. Because we know the expected value for VE(t) (vaccine effectiveness over time) for each simulation, we can calculate the root mean square error (RMSE) for the time-stratified model against the expected value, but in a real-world study this would not be the case. However, over a variety of R₀ values, we find that the minimum Akaike information criterion (AIC), shown as x’s in A–C, corresponds well to where low RMSE is found. Thus, despite the very different dynamics in each of these systems, a simple check over multiple minimums for events and days for minimum AIC should yield a good result.

Figure 6.

Optimizing the time categories gives an excellent visualization of the true form. In (A) simple 100-event partitions are used; in (B) partitions are a minimum of 6 days with 700 events, per the minimum Akaike information criterion for this simulation. This offers an improvement to the estimate even in such an extreme case using a relatively simple method. Optimizing the partitioning lowered the root mean square error from 4.33% to 2.85%. Abbreviations: SR, Schoenfeld residuals method; TVI, time-vaccine interaction method; VE, vaccine effectiveness.