Combining probabilistic forecasts of COVID-19 mortality in the United States

Abstract

The COVID-19 pandemic has placed forecasting models at the forefront of health policy making. Predictions of mortality, cases and hospitalisations help governments meet planning and resource allocation challenges. In this paper, we consider the weekly forecasting of the cumulative mortality due to COVID-19 at the national and state level in the U.S. Optimal decision-making requires a forecast of a probability distribution, rather than just a single point forecast. Interval forecasts are also important, as they can support decision making and provide situational awareness. We consider the case where probabilistic forecasts have been provided by multiple forecasting teams, and we combine the forecasts to extract the wisdom of the crowd. We use a dataset that has been made publicly available from the COVID-19 Forecast Hub. A notable feature of the dataset is that the availability of forecasts from participating teams varies greatly across the 40 weeks in our study. We evaluate the accuracy of combining methods that have been previously proposed for interval forecasts and predictions of probability distributions. These include the use of the simple average, the median, and trimming methods. In addition, we propose several new weighted combining methods. Our results show that, although the median was very useful for the early weeks of the pandemic, the simple average was preferable thereafter, and that, as a history of forecast accuracy accumulates, the best results can be produced by a weighted combining method that uses weights that are inversely proportional to the historical accuracy of the individual forecasting teams.

Keywords: COVID-19; Distributional forecasts; Forecast combining; Interval forecasts; OR in health services.

Fig. 1

Number of reported COVID-19 deaths in the U.S. up to 30 January 2021.

Fig. 2

Rise in COVID-19 deaths in the five states with highest cumulative total on 30 January 2021.

Fig. 3

One week-ahead distributional forecasts produced with Week 28 as forecast origin.

Fig. 4

Number of forecasting teams included in our study for each forecast origin. The stacked bars indicate the split between teams using compartmental models, and alternatives.

Fig. 5

Timeline showing whether forecasts from each team were included in our study for each forecast origin. The circles indicate the number of the 52 series for which forecasts were available and eligible. Black, grey, white and small black circles indicate: all 52 time series, either 51 or 50, between 49 and 26, and 25 or fewer, respectively.

Fig. 6

Calibration hit percentages for bounds on 50% and 95% intervals, computed using the 30-week out-of-sample period.

Fig. 7

Comparison of the LQS for the simple average and inverse LQS combining methods for each of the 52 mortality series. LQS averaged over the 30-week out-of-sample period.

Fig. 8

Calibration of distributional forecasts assessed using hit percentages for the 23 quantile probability levels θ. Hit percentages computed using the 30-week out-of-sample period.

Fig. 9

LQS for distributional forecasts from simple average combinations of different numbers of forecasts. LQS averaged over the 30-week out-of-sample period.