The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt

Abstract

Infectious disease forecasting is gaining traction in the public health community; however, limited systematic comparisons of model performance exist. Here we present the results of a synthetic forecasting challenge inspired by the West African Ebola crisis in 2014-2015 and involving 16 international academic teams and US government agencies, and compare the predictive performance of 8 independent modeling approaches. Challenge participants were invited to predict 140 epidemiological targets across 5 different time points of 4 synthetic Ebola outbreaks, each involving different levels of interventions and "fog of war" in outbreak data made available for predictions. Prediction targets included 1-4 week-ahead case incidences, outbreak size, peak timing, and several natural history parameters. With respect to weekly case incidence targets, ensemble predictions based on a Bayesian average of the 8 participating models outperformed any individual model and did substantially better than a null auto-regressive model. There was no relationship between model complexity and prediction accuracy; however, the top performing models for short-term weekly incidence were reactive models with few parameters, fitted to a short and recent part of the outbreak. Individual model outputs and ensemble predictions improved with data accuracy and availability; by the second time point, just before the peak of the epidemic, estimates of final size were within 20% of the target. The 4th challenge scenario - mirroring an uncontrolled Ebola outbreak with substantial data reporting noise - was poorly predicted by all modeling teams. Overall, this synthetic forecasting challenge provided a deep understanding of model performance under controlled data and epidemiological conditions. We recommend such "peace time" forecasting challenges as key elements to improve coordination and inspire collaboration between modeling groups ahead of the next pandemic threat, and to assess model forecasting accuracy for a variety of known and hypothetical pathogens.

Keywords: Data accuracy; Ebola epidemic; Forecasting challenge; Mathematical modeling; Model comparison; Prediction horizon; Prediction performance; Synthetic data.

Figure 1

Example of ensemble prediction graph provided to participants throughout the challenge. Here for prediction time point 5. The grey area represents the cone of incidence predictions 1–4 weeks ahead (min and max across all teams) while the red line is the mean. The black dotted line represents the synthetic epidemic curve.

Figure 2

Performance statistics for incidence forecasts, displaying data for all prediction time points. Top: Box plots of the mean absolute error by team, across all scenarios. Red indicates the Bayesian ensemble mean (smallest absolute error). Bottom: Agreement between synthetic and predicted incidences by team for data-rich scenario 1.

Figure 3

Longer-term prediction targets for data-rich scenario 1. The first 5 panels represent the timing and magnitude of predicted Ebola peaks by team and prediction time point. The grey curve represents the target outbreak incidence data, with dark grey representing the amount of data available for prediction at each time point, while the light gray curve displays the full outbreak. The bottom right panel represents the distribution of final size predictions across teams by prediction time point. The solid horizontal grey line marks the true final size of the outbreak in scenario 1.