Neural network models for influenza forecasting with associated uncertainty using Web search activity trends

Abstract

Influenza affects millions of people every year. It causes a considerable amount of medical visits and hospitalisations as well as hundreds of thousands of deaths. Forecasting influenza prevalence with good accuracy can significantly help public health agencies to timely react to seasonal or novel strain epidemics. Although significant progress has been made, influenza forecasting remains a challenging modelling task. In this paper, we propose a methodological framework that improves over the state-of-the-art forecasting accuracy of influenza-like illness (ILI) rates in the United States. We achieve this by using Web search activity time series in conjunction with historical ILI rates as observations for training neural network (NN) architectures. The proposed models incorporate Bayesian layers to produce associated uncertainty intervals to their forecast estimates, positioning themselves as legitimate complementary solutions to more conventional approaches. The best performing NN, referred to as the iterative recurrent neural network (IRNN) architecture, reduces mean absolute error by 10.3% and improves skill by 17.1% on average in nowcasting and forecasting tasks across 4 consecutive flu seasons.

Fig 1. Negative log-likelihood (NLL) and mean absolute error (MAE) for each NN model averaged over all four test flu seasons (2015/16 to 2018/19).

Scores for different forecast horizons (γ) are shown. Lower values are better. We also provide a comparison with IRNN trained without using any Web search activity data (IRNN₀), and a simple persistence model (PER). Note that NLL cannot be determined for PER as it does not provide an associated uncertainty. S1 Fig shows the results for all metrics.

Fig 2. IRNN forecasts for all 4 test seasons (2015/16 to 2018/19) and forecasting horizons (γ = 7, 14, 21, and 28).

Confidence intervals (uncertainty estimates) are shown at 50% and 90% levels, and are visually distinguished by darker and lighter colour overlays respectively. The influenza-like illness (ILI) rate (ground truth) is shown by the black line.

Fig 3. Calibration plots for the forecasts made by the three NN models (FF, SRNN, and IRNN) averaged over the four test periods (2015/16 to 2018/19) and shown for the 4 forecasting horizons (γ).

The lines show how frequently the ground truth falls within a confidence interval (CI) of the same level. To be more precise, a point (x, y) denotes that the proportion y ∈ [0, 1] of the forecasts when combined with a CI at the x × 100% level include the ground truth (successful forecasts). The optimal calibration is shown by the diagonal black line. Points above or below the diagonal indicate an over- or under-estimation of uncertainty, and hence an under- or over-confident model, respectively. The shadows show the upper and lower quartile of the calibration curves when the models are trained multiple times with different initialisation seeds. The plot broken out into separate test periods is shown in the Supporting Information (S11 Fig).

Fig 4. Diagram of the IRNN architecture where for the recurrent layers (RNN) we have used a Gated Recurrent Unit.

An ILI rate, F ∈ [0, 1], and m search query frequencies, $\mathbf{Q}\in {\mathbb{R}}_{\ge 0}^m$, beginning from time point (day) t₀ − τ are fed into the network a day at a time. τ denotes the window size of past observations that we consider (τ + 1 = 56 days). The reporting delay of the ILI rates means that when an ILI rates are available up to day t₀, search query frequencies are available up to day t₀ + δ, where δ = 14 days in our experiments. Dashed arrow lines denote that the model is called for multiple time-steps (where a time step is a day). For days t₀ − τ to t₀, IRNN enters a warm-up phase where it sets the hidden states in the RNN layer without making any predictions. For days t₀ to t₀ + δ, we can observe search query frequencies, but we cannot observe ILI rates. At this stage, IRNN performs nowcasting with respect to input Q. During nowcasting the estimated ILI rate ${\widehat{F}}_t$ is combined with the true search frequencies Q_t use as the input for the next time step. The query search frequency estimates which are not used (as they are known to us) are shown by a faded box. For days t₀ + δ + 1 to t₀ + γ, where γ denotes the forecasting horizon, IRNN conducts pure forecasting as neither search query frequencies nor ILI rates are known for that period. Forecasted values for both of them are used as inputs for subsequent time steps. The full sequence of both predicted ILI rates and search query frequencies is used in the training loss.