Scenario design for infectious disease projections: Integrating concepts from decision analysis and experimental design

Abstract

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, situational awareness, horizon scanning, forecasting, and value of information) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Keywords: Design of experiments; Multi-model projections; Scenario modeling; Sensitivity analysis; Value of information.

Fig. 1.

Graphical depiction of six classes of scenario design, with heuristic examples. (A) In a Decision Making scenario design, the axis or axes are variables that are under the control of the decision maker; the purpose of the design is to understand the outcomes associated with different interventions. (B) Four classes of design have a similar structure (uncertainty axes only) but different purposes. (B1) Sensitivity Analysis designs focus on understanding the role of different sources of uncertainty on the outcomes of interest. (B2) Situational Awareness designs resemble Sensitivity Analysis designs, but have an additional purpose to provide insight about potential outcomes that may be relevant for ancillary decisions. (B3) Horizon Scanning designs explore the edges of the epistemic uncertainty, often to prompt insights about what could happen in the future, in an effort to develop new interventions. (B4) Forecasting designs postulate multiple hypotheses in the parameter space, with an appropriately weighted average of outcomes constituting a well-calibrated forecast, given the current uncertainty. (C) A Value of Information (VOI) design (decision axis×uncertainty axis) examines whether a source of uncertainty affects the relative effects of interventions. The dashed lines represent the current point estimates and the shaded regions represent the current confidence intervals for the uncertainty parameters.

Fig. 2.

Overview of scenario design process. First, determine the purpose of the scenario modeling exercise, including the questions to be addressed and the intended audience. This purpose informs all other design decisions. The taxonomy defined in Fig. 1 should be applied at this step. Then, define the features that distinguish scenarios and those that are common across scenarios. Last, consider other design issues that may be relevant during all phases of scenario design.