Real-time influenza forecasts during the 2012-2013 season

Abstract

Recently, we developed a seasonal influenza prediction system that uses an advanced data assimilation technique and real-time estimates of influenza incidence to optimize and initialize a population-based mathematical model of influenza transmission dynamics. This system was used to generate and evaluate retrospective forecasts of influenza peak timing in New York City. Here we present weekly forecasts of seasonal influenza developed and run in real time for 108 cities in the USA during the recent 2012-2013 season. Reliable ensemble forecasts of influenza outbreak peak timing with leads of up to 9 weeks were produced. Forecast accuracy increased as the season progressed, and the forecasts significantly outperformed alternate, analogue prediction methods. By week 52, prior to peak for the majority of cities, 63% of all ensemble forecasts were accurate. To our knowledge, this is the first time predictions of seasonal influenza have been made in real time and with demonstrated accuracy.

Figure 1

Calibration of forecast accuracy as a function of ensemble spread. Retrospective forecasts of outbreak peak timing initiated for each of the 2003–2004 through 2011–2012 seasons, excluding the pandemic seasons of 2008–2009 and 2009–2010. Retrospective forecasts were made for 115 cities, which were then aggregated by census division or nationally. Plots present the probability that an ensemble predicted mode peak timing is accurate within +/− 1 week of the observed ILI+ peak as a function of ensemble predicted peak timing variance log transformed. A) Training and forecast made using climatological AH, census division aggregation; B) As in A), but aggregated nationally. The colored lines are for ensemble mode peak predictions 10+ weeks in the future (magenta), 7–9 weeks in the future (blue), 4–6 weeks in the future (cyan), 1–3 weeks in the future (red), 0–2 weeks in the past (green), and 3–5 weeks in the past (black).

Figure 1

Calibration of forecast accuracy as a function of ensemble spread. Retrospective forecasts of outbreak peak timing initiated for each of the 2003–2004 through 2011–2012 seasons, excluding the pandemic seasons of 2008–2009 and 2009–2010. Retrospective forecasts were made for 115 cities, which were then aggregated by census division or nationally. Plots present the probability that an ensemble predicted mode peak timing is accurate within +/− 1 week of the observed ILI+ peak as a function of ensemble predicted peak timing variance log transformed. A) Training and forecast made using climatological AH, census division aggregation; B) As in A), but aggregated nationally. The colored lines are for ensemble mode peak predictions 10+ weeks in the future (magenta), 7–9 weeks in the future (blue), 4–6 weeks in the future (cyan), 1–3 weeks in the future (red), 0–2 weeks in the past (green), and 3–5 weeks in the past (black).

Figure 2

Accuracy of 2012–2013 real-time forecasts. Plots comparing the weekly fraction of accurate SIRS-EAKF forecasts with the accuracy of analog forecasts derived from historical probabilities (see Supplementary Methods). Top) Weekly SIRS-EAKF forecast accuracy and resampled analog predictions using two alternate resampling approaches. Bottom) Weekly SIRS-EAKF forecast accuracy and resampled analog predictions based on historically observed durations between initially elevated ILI+ and peak ILI+. Only cities that have exceeded an onset, or initial threshold, level of elevated ILI+ are included in the analog forecast for a given week. Three different onset thresholds are shown: 100, 500 and 2000 ILI+. For all the analog forecasts, the thick line depicts the mean fraction of accurate forecasts while the shading and thin lines delineate 95% bootstrap confidence intervals.

Figure 3

Expected accuracy of peak timing forecasts for the 2012–2013 season. Week 47-8 forecasts were made for 108 cities, which were then aggregated nationally. Plots present the probability that an ensemble predicted mode peak timing is accurate within +/− 1 week of the observed ILI+ peak as a function of ensemble predicted peak timing variance log transformed. The blue lines are the 2012–2013 predictions grouped by forecast lead; the red lines are the expected accuracy based on the retrospective forecasts also aggregated nationally (as shown in Figure 1B).

Figure 4

Weekly predictions of CDC ILI+ and GFT ILI+ peak timing for HHS regions. Top) The fraction of all SIRS-EAKF forecasts each week (made using SIRS model without AH modulation of R₀); the Week Weeks 1–6 forecasts were run in near real time; the Week 47–52 forecasts were run using data downloaded following Week 1. Middle) Plots of observed HHS CDC ILI+ as reported through Week 12, 2013; using this metric, all HHS peak during Week 52, except HHS Region 9, which peaked during Week 4. Bottom) Plots of observed HHS GFT ILI+ as reported through Week 12, 2013; 9 of the 10 GFT ILI+ HHS regions peak later than their counterpart CDC ILI+ estimate. From Week 40 (2012) through Week 12 (2013), HHS GFT ILI+ was on average 1.61 times corresponding estimates of HHS CDC ILI+, and during Weeks 1–5 HHS GFT ILI+ was on average 2.20 times HHS CDC ILI+.