An expert judgment model to predict early stages of the COVID-19 pandemic in the United States

Abstract

From February to May 2020, experts in the modeling of infectious disease provided quantitative predictions and estimates of trends in the emerging COVID-19 pandemic in a series of 13 surveys. Data on existing transmission patterns were sparse when the pandemic began, but experts synthesized information available to them to provide quantitative, judgment-based assessments of the current and future state of the pandemic. We aggregated expert predictions into a single "linear pool" by taking an equally weighted average of their probabilistic statements. At a time when few computational models made public estimates or predictions about the pandemic, expert judgment provided (a) falsifiable predictions of short- and long-term pandemic outcomes related to reported COVID-19 cases, hospitalizations, and deaths, (b) estimates of latent viral transmission, and (c) counterfactual assessments of pandemic trajectories under different scenarios. The linear pool approach of aggregating expert predictions provided more consistently accurate predictions than any individual expert, although the predictive accuracy of a linear pool rarely provided the most accurate prediction. This work highlights the importance that an expert linear pool could play in flexibly assessing a wide array of risks early in future emerging outbreaks, especially in settings where available data cannot yet support data-driven computational modeling.

Fig 1. Expert predictions of confirmed COVID-19 cases and deaths.

(A.) Expert linear pool predictions of the total number of deaths by the end of 2020 from five surveys asked between March 16 and May 4, 2020. Points show the median estimate. Bars show 90% prediction intervals for the first four surveys and an 80% prediction interval for the fifth survey. The dotted line is the reported total number of deaths by The COVID Tracking Project as of December 31, 2020. (B.) Expert linear pool forecasts, made on Monday and Tuesdays, of the number of cases to be reported by the end of the week (Sunday, date shown on x-axis) from thirteen surveys administered between February 23 and May 17, 2020. The first eight surveys asked experts to provide smallest, most likely, and largest possible values for the number of confirmed cases, and the last five asked experts to assign probabilities to ranges of values for confirmed cases. Light blue points represent the median of the expert linear pool distribution. Dark points represent the eventually observed value reported by The COVID Tracking Project. Prediction intervals at the 90% level are shown in shaded blue bars. The 90% prediction intervals included the true number of cases in all thirteen forecasts.

Fig 2. Expert predictions of total number of SARS-CoV-2 infections.

Estimates of the total number of infections with the SARS-CoV-2 virus, made in different weeks across early 2020. (A.) Estimates of the total number of SARS-CoV-2 infections (both observed and unobserved) from the expert linear pool model (green dots, with 80% prediction intervals), four contemporaneous estimates (blue, red, yellow, and purple bars), and four retrospective estimates from computational models (fit in 2021) [–, –60]. Prediction intervals at the 80% level are shown for all prospective estimates except for the Bedford estimate which provided a “best guess” prediction interval and a second interval double the size of the first shown as a narrower line [56]. Expert predictions aligned with contemporaneous model estimates throughout the entire survey time. Real-time estimates from both models and expert linear pools were 2–3 orders of magnitude smaller than retrospective model estimates (estimates generated in early 2021) for early March 2020 and more in alignment by the end of April 2020. (B.) Expert linear pool distributions of the fraction of all infections reported as confirmed cases. In the first three surveys, experts provided a predicted percent of infections that had been confirmed as cases by laboratory test, in the next four (dates in boldface) they directly estimated the total number of infections. Surveys 4–6 asked experts to provide the smallest, most likely, and highest number of total infections, and the last survey asked experts to provide a 10th, 50th, and 90th percentile (shadings not included in the figure).

Fig 3. Counterfactual predictions of reported COVID-19 cases.

Summaries of counterfactual predictions of reported COVID-19 cases made by experts for three states. In each panel, experts made predictions under an “optimistic” and “pessimistic” scenario about the impact of re-opening on COVID-19 transmission. All predictions were made about an outcome between 3–5 weeks into the future. The time at which predictions were made are shown with a black dot. The scenario that ended up being more aligned with reality at the target prediction date are shown in blue. The date at which relevant policies were enacted between the survey date and the resolution date are indicated with a vertical dashed line. (A). An expert linear pool median and 80% CI (10th percentile and 90th percentile) made on April 27, 2020 (black circle) of the 7- day moving average of reported COVID-19 cases for the state of Georgia for the week of May 10 to May 16, 2020 under two scenarios: if the state of Georgia reopens several businesses or “loosens restrictions” (orange) or if restriction were not loosened (blue). The 7-day moving average reported by the Georgia Department of Health is in black. (B). Expert linear pool predictions made on May 4, 2020 of the 7-day moving average of reported COVID-19 cases for the state of Texas for the week ending on June 13, 2020. Predictions were made under the differing assumptions that (i) Texas continued to loosen restrictions (blue) and (ii) that the state did not loosen restrictions (orange). The moving average reported by the Texas Department of Health is shown as the solid black line. Quantiles for the expert linear pool predictions were imputed by assuming a uniform distribution over values within 5 intervals. (C) Expert linear pool predictions made on May 11, 2020 of the 7-day moving average of reported COVID-19 cases for the state of Washington for the week ending on June 7, 2020. Predictions were made under two assumptions: (i) Washington would begin their Phase II plan on May 16, 2020, an accelerated restart, for all counties (orange) or Phase II would not begin by May 16 for all counties (blue). The moving average reported by the Washington Department of Health is shown as the solid black line.

Fig 4. Forecast accuracy for expert predictions of cases and deaths.

Evaluation of forecast accuracy for forecasts of cumulative COVID-19 deaths (A) and cases (B). For both types of questions, the methods used to elicit probabilistic forecasts changed and this point is indicated by a vertical dashed line. Predictions are shown from each expert (light dots), the median expert (dark diamond), and the linear pool (dark square) compared to an “unskilled” forecaster (see Methods). Higher relative forecast skill indicates better performance than an unskilled forecaster and a zero relative forecast skill represents identical performance with an unskilled forecaster. (A). Relative forecast skill of the cumulative number of COVID-19 deaths by December 31, 2020 (see Fig 1A). Over 50% of experts made better predictions of year-end COVID-19 deaths than an unskilled forecaster on each of the five occasions this question was asked. Experts’ median relative forecast skill was higher than the linear pool forecast skill for the latest prediction of year-end deaths when asked to provide percentiles compared to the smallest, most likely, and largest number of deaths. (B.) Relative forecast skill of the number of cases to be reported by the end of the week from thirteen surveys administered between February 23 and May 17, 2020. Individual experts’ accuracy was mixed with some experts performing better than an unskilled forecaster and others scoring worse. In the first five surveys, the median expert made less skilled forecasts than the unskilled forecaster. Experts’ median relative forecast skill was smaller than the linear pool forecast skill when asked to provide a smallest, most likely, and largest number of cases and similar to a linear pool when asked to assign probabilities to a set of intervals where the true number of cases could fall.

Fig 5. Expert analytic versus intuitive thinking.

Experts’ self-assessed percentage of analytic vs. intuitive thinking when making predictions, reporting 0 when an expert uses only their intuition and 100 when they relied solely on models and experience. To make predictions over a wide variety of targets, experts reported a mixture of using models/experience and intuition, with the median expert claiming to rely 75% on experience.

Fig 6. Expert linear pool forecast skill.

The forecast skill percentile of linear pool predictions compared to individual expert predictions across all 13 surveys (vertical axis) differentiated by the type of target. A linear pool of expert judgment often scores in the top 50th percentile independent of the type of question (B.) The mean, median, 25th, and 75th percentile for forecast skill percentile for all individual experts who completed 10 or more surveys (blue) and for the linear pool (red). Compared to individual experts, a linear pool has the highest mean and highest 25th percentile forecast skill percentile. (C.) The median forecast skill percentile across surveys for experts (blue) and the linear pool (red). Over time the linear pool median forecast skill percentile is above 0.50 for all but one survey and for five surveys the linear pool generated the most accurate predictions.