Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
[Preprint]. 2023 Dec 11:2023.12.08.23299726.
doi: 10.1101/2023.12.08.23299726.

Evaluation of FluSight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations

Sarabeth M Mathis  1 , Alexander E Webber  1 , Tomás M León  2 , Erin L Murray  2 , Monica Sun  2 , Lauren A White  2 , Logan C Brooks  3   4 , Alden Green  3 , Addison J Hu  3 , Daniel J McDonald  5 , Roni Rosenfeld  3 , Dmitry Shemetov  3 , Ryan J Tibshirani  3   4 , Sasikiran Kandula  6 , Sen Pei  7 , Jeffrey Shaman  7   8 , Rami Yaari  7 , Teresa K Yamana  7 , Pulak Agarwal  9 , Srikar Balusu  9 , Gautham Gururajan  9 , Harshavardhan Kamarthi  9 , B Aditya Prakash  9 , Rishi Raman  9 , Alexander Rodríguez  10 , Zhiyuan Zhao  9 , Akilan Meiyappan  11 , Shalina Omar  11 , Prasith Baccam  12 , Heidi L Gurung  12 , Steve A Stage  13 , Brad T Suchoski  12 , Marco Ajelli  14 , Allisandra G Kummer  14 , Maria Litvinova  14 , Paulo C Ventura  14 , Spencer Wadsworth  15 , Jarad Niemi  15 , Erica Carcelen  16 , Alison L Hill  16 , Sung-Mok Jung  17 , Joseph C Lemaitre  17 , Justin Lessler  17 , Sara L Loo  16 , Clifton D McKee  16 , Koji Sato  16 , Claire Smith  16 , Shaun Truelove  16 , Thomas McAndrew  18 , Wenxuan Ye  18 , Nikos Bosse  19 , William S Hlavacek  20 , Yen Ting Lin  20 , Abhishek Mallela  20 , Ye Chen  21 , Shelby M Lamm  21 , Jaechoul Lee  21 , Richard G Posner  21 , Amanda C Perofsky  22 , Cécile Viboud  22 , Leonardo Clemente  23 , Fred Lu  23 , Austin G Meyer  23 , Mauricio Santillana  23 , Matteo Chinazzi  23 , Jessica T Davis  23 , Kunpeng Mu  23 , Ana Pastore Y Piontti  23 , Alessandro Vespignani  23 , Xinyue Xiong  23 , Michal Ben-Nun  24 , Pete Riley  24 , James Turtle  24 , Chis Hulme-Lowe  25 , Shakeel Jessa  25 , V P Nagraj  26 , Stephen D Turner  26 , Desiree Williams  26 , Avranil Basu  27 , John M Drake  27 , Spencer J Fox  28 , Graham C Gibson  20 , Ehsan Suez  28 , Edward W Thommes  29   30 , Monica G Cojocaru  29 , Estee Y Cramer  31 , Aaron Gerding  31 , Ariane Stark  31 , Evan L Ray  31 , Nicholas G Reich  31 , Li Shandross  31 , Nutcha Wattanachit  31 , Yijin Wang  31 , Martha W Zorn  31 , Majd Al Aawar  32 , Ajitesh Srivastava  32 , Lauren A Meyers  33 , Aniruddha Adiga  34 , Benjamin Hurt  34 , Gursharn Kaur  34 , Bryan L Lewis  34 , Madhav Marathe  34 , Srinivasan Venkatramanan  34 , Patrick Butler  35 , Andrew Farabow  35 , Nikhil Muralidhar  36 , Naren Ramakrishnan  35 , Carrie Reed  1 , Matthew Biggerstaff  1 , Rebecca K Borchering  1
Affiliations

Evaluation of FluSight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations

Sarabeth M Mathis et al. medRxiv. .

Update in

  • Title evaluation of FluSight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations.
    Mathis SM, Webber AE, León TM, Murray EL, Sun M, White LA, Brooks LC, Green A, Hu AJ, Rosenfeld R, Shemetov D, Tibshirani RJ, McDonald DJ, Kandula S, Pei S, Yaari R, Yamana TK, Shaman J, Agarwal P, Balusu S, Gururajan G, Kamarthi H, Prakash BA, Raman R, Zhao Z, Rodríguez A, Meiyappan A, Omar S, Baccam P, Gurung HL, Suchoski BT, Stage SA, Ajelli M, Kummer AG, Litvinova M, Ventura PC, Wadsworth S, Niemi J, Carcelen E, Hill AL, Loo SL, McKee CD, Sato K, Smith C, Truelove S, Jung SM, Lemaitre JC, Lessler J, McAndrew T, Ye W, Bosse N, Hlavacek WS, Lin YT, Mallela A, Gibson GC, Chen Y, Lamm SM, Lee J, Posner RG, Perofsky AC, Viboud C, Clemente L, Lu F, Meyer AG, Santillana M, Chinazzi M, Davis JT, Mu K, Pastore Y Piontti A, Vespignani A, Xiong X, Ben-Nun M, Riley P, Turtle J, Hulme-Lowe C, Jessa S, Nagraj VP, Turner SD, Williams D, Basu A, Drake JM, Fox SJ, Suez E, Cojocaru MG, Thommes EW, Cramer EY, Gerding A, Stark A, Ray EL, Reich NG, Shandross L, Wattanachit N, Wang Y, Zorn MW, Aawar MA, Srivastava A, Meyers LA, Adiga A, Hurt B, Kaur G, Lewis BL, Marathe M, Venkatramanan S, Butler P, Farabow A, Ramakrishnan N, Muralidhar N, Reed C, Biggerstaff M, Borchering RK. Mathis SM, et al. Nat Commun. 2024 Jul 26;15(1):6289. doi: 10.1038/s41467-024-50601-9. Nat Commun. 2024. PMID: 39060259 Free PMC article.

Abstract

Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. Forecasting teams were asked to provide national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one through four weeks ahead for the 2021-22 and 2022-23 influenza seasons. Across both seasons, 26 teams submitted forecasts, with the submitting teams varying between seasons. Forecast skill was evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperformed the baseline model across forecast weeks and locations in 2021-22 and 12 out of 18 models in 2022-23. Averaging across all forecast targets, the FluSight ensemble was the 2nd most accurate model measured by WIS in 2021-22 and the 5th most accurate in the 2022-23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degraded over longer forecast horizons and during periods of rapid change. Current influenza forecasting efforts help inform situational awareness, but research is needed to address limitations, including decreased performance during periods of changing epidemic dynamics.

PubMed Disclaimer

Conflict of interest statement

Competing interests: E.W.T. is an employee of Sanofi, which manufactures influenza vaccines. J.S. and Columbia University disclose partial ownership of SK Analytics. J.S. discloses consulting for BNI.

Figures

Figure 1:
Figure 1:
National weekly observed hospitalizations (black points) along with FluSight ensemble forecasts for four weeks of submissions in the 2021–22 season (panel a) and seven weeks of submissions in the 2022–23 season (panel b). The median FluSight ensemble forecast values (blue points) are shown with the corresponding 50%, 80%, and 95% prediction intervals (blue shaded regions).
Figure 1:
Figure 1:
National weekly observed hospitalizations (black points) along with FluSight ensemble forecasts for four weeks of submissions in the 2021–22 season (panel a) and seven weeks of submissions in the 2022–23 season (panel b). The median FluSight ensemble forecast values (blue points) are shown with the corresponding 50%, 80%, and 95% prediction intervals (blue shaded regions).
Figure 2:
Figure 2:
Standardized rank of weighted interval score (WIS) over all forecast jurisdictions and horizons (1- to 4-week ahead), for the FluSight ensemble and each team submitting at least 75% of the forecast targets (see Table 1 for qualifying teams and season metrics).
Figure 3:
Figure 3:
State-level WIS values for each team relative to the FluSight baseline model. The range of Relative WIS values below 1, in blue, indicate better performance than the FluSight baseline (white). Relative WIS values above 1, in red, indicate poor performance relative to the FluSight baseline. Teams are ordered on horizontal axis from lowest to highest Relative WIS values for each season. Analogous jurisdiction-specific relative WIS scores on log transformed counts are displayed in Figure S7.
Figure 4:
Figure 4:
Time series of log transformed absolute WIS (panel a) and 1- and 4-week ahead 95% coverage (panel b) for state and territory targets. Note that the forecast evaluation period translates to 1-week ahead forecast target end dates from February 26 to June 25, 2022, and October 22, 2022, to May 20, 2023, and 4-week ahead forecast target end dates from March 19 to July 16, 2022, and November 5, 2022, to June 10, 2023. Weekly results for the FluSight baseline and ensemble models are shown in red and blue respectively. Results for individual contributing models are shown in light gray.
See this image and copyright information in PMC

References

    1. Weekly U.S. Influenza Surveillance Report. https://www.cdc.gov/flu/weekly/index.htm. (2023)
    1. U.S. Outpatient Influenza-like Illness Surveillance Network (ILINet). https://wwwn.cdc.gov/ILINet/. (2023)
    1. Influenza Hospitalization Surveillance Network (FluSurv-NET). https://www.cdc.gov/flu/weekly/influenza-hospitalization-surveillance.htm. (2023)
    1. Lutz C.S., et al. Applying infectious disease forecasting to public health: a path forward using influenza forecasting examples. BMC Public Health, 19–1659 (2019). - PMC - PubMed
    1. McGowan C. J., et al.; Influenza forecasting working group, Collaborative efforts to forecast seasonal influenza in the United States, 2015–2016. Sci. Rep. 9, 683 (2019). - PMC - PubMed

Publication types