Multiple models for outbreak decision support in the face of uncertainty

Abstract

Policymakers must make management decisions despite incomplete knowledge and conflicting model projections. Little guidance exists for the rapid, representative, and unbiased collection of policy-relevant scientific input from independent modeling teams. Integrating approaches from decision analysis, expert judgment, and model aggregation, we convened multiple modeling teams to evaluate COVID-19 reopening strategies for a mid-sized United States county early in the pandemic. Projections from seventeen distinct models were inconsistent in magnitude but highly consistent in ranking interventions. The 6-mo-ahead aggregate projections were well in line with observed outbreaks in mid-sized US counties. The aggregate results showed that up to half the population could be infected with full workplace reopening, while workplace restrictions reduced median cumulative infections by 82%. Rankings of interventions were consistent across public health objectives, but there was a strong trade-off between public health outcomes and duration of workplace closures, and no win-win intermediate reopening strategies were identified. Between-model variation was high; the aggregate results thus provide valuable risk quantification for decision making. This approach can be applied to the evaluation of management interventions in any setting where models are used to inform decision making. This case study demonstrated the utility of our approach and was one of several multimodel efforts that laid the groundwork for the COVID-19 Scenario Modeling Hub, which has provided multiple rounds of real-time scenario projections for situational awareness and decision making to the Centers for Disease Control and Prevention since December 2020.

Keywords: cognitive biases; decision theory; multi-model aggregation.

Fig. 1.

Aggregate and 17 individual model results for target objective and intervention scenario pairs. Medians, 50% prediction intervals (PIs, the range of values within which we project future outcomes will fall with 50% probability), and 90% PIs are indicated as points, thick lines, and thin lines, respectively. Colors denote ranking of each intervention for a single objective, where dark blue signifies the lowest value (best performance) and dark red signifies the highest value (worst performance). Ties in ranks are colored as intermediate values. Ties between ranks 1 and 2 and ranks 3 and 4 are shown as an intermediate blue and red, respectively; yellow indicates a tie in ranks across all interventions. The five panels show the results for: i) cumulative SARS-CoV-2 infections (rather than reported cases) between May 15 and November 15, 2020; ii) cumulative deaths due to COVID-19 over the same period, with an inset displaying the results for a smaller range of values, beginning with zero and containing the 50% prediction intervals; iii) the peak number of hospitalizations over the same period, with the inset showing a smaller range of values, and the hospital capacity of 200 beds with a vertical dotted line; iv) the probability of an outbreak of greater than 10 new cases per day after May 15; and v) the number of days that nonessential workplaces are closed between May 15 and November 15. The interventions include “closed”, workplace closure throughout the 6-mo period; “5-percent”, nonessential workplace reopening when cases decline below 5% of the peak caseload; “2-wk”, nonessential workplace reopening 2 wk after the peak; and “open”, immediate reopening of all workplaces on May 15. The setting is a generic US county of 100,000 people that had experienced 180 reported cases and 6 deaths as of May 15, 2020; all schools are assumed to be closed throughout the projection period. (A) Aggregate distributions for each objective and intervention scenario pair. The aggregate distributions were calculated as the unweighted average of the individual cumulative distribution functions across the 16 modeling groups. (B) Individual model results for each objective and intervention scenario pair. Results are presented here essentially as they were shown to modeling teams during the between-round discussion (except each projection was labeled with a letter that was only shared with the team that generated that projection); projections were intentionally presented anonymously to avoid groupthink and authority bias.

Fig. 2.

Resolution of linguistic uncertainty in the discussion following round 1 of modeling about the number of days nonessential workplaces are closed. Direct comparison of round 1 and 2 results for days closed for the fully closed intervention. In round 1, groups used a variety of start dates (start of the forecast, first day of stay at home orders, or state of emergency declarations) and one group implemented a weighting for essential and nonessential business closures and associated compliance issues explicitly.

Fig. 3.

Comparison of aggregate reported county death data to modeled deaths for the closed intervention. Boxplot of cumulative reported deaths from 84 US counties with full or partial stay at home orders in place from May 15 to November 15, 2020 (median deaths: 48; 50% IQR: 27, 71) and model results for cumulative deaths from May 15 to November 15, 2020 under the closed intervention (median deaths: 73; 50% IQR: 12, 228). Vertical line shows median value, box shows IQR (25th to 75th quantiles), and whiskers extend to the 5th and 95th quantiles. Inset shows overlap of box area for the plots.

Fig. 4.

Overview of the MMODS approach: Embracing uncertainty to inform COVID-19 nonessential workplace reopening decisions using multiple models for outbreak decision support. Aggregate results ranked interventions consistently for specific objectives (e.g., minimizing cumulative infections) (A) even though the model-specific magnitude of projections varied greatly (B). Ensemble aggregate distributions quantify risk for decision makers and enable them to inform decisions in the context of resource limitations, such as the availability of hospital beds denoted by the vertical dotted line (C). An intermediate discussion between two rounds of independent model projections resolved inconsistencies in interpretation of intervention implementation as well as objective specification (D).