Scenario Design for Infectious Disease Projections: Integrating Concepts from Decision Analysis and Experimental Design

Abstract

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, value of information, situational awareness, horizon scanning, and forecasting) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Keywords: Scenario modeling; design of experiments; multi-model projections; sensitivity analysis; value of information.

Figure 1.

Graphical depiction of three classes of scenario designs, with heuristic examples. (A) In a Decision Making scenario design, the axis or axes are variables that are under the control of the decision maker; the purpose of the design is to understand the outcomes associated with different interventions. (B) Sensitivity Analysis designs focus on understanding the role of different sources of uncertainty on the outcomes of interest. (C) A Value of Information (VOI) design (decision axis x uncertainty axis) examines whether a source of uncertainty affects the relative effects of interventions. The shaded regions represent the current confidence intervals for the uncertainty parameters.

Figure 2.

Scenario design classes related to Sensitivity Analysis designs (uncertainty x uncertainty) that have decision-adjacent purposes. (A) Situational Awareness designs may appear indistinguishable from Sensitivity Analysis designs, but have an additional purpose to provide insight about potential outcomes that may be relevant for ancillary decisions. (B) Horizon Scanning designs explore the edges of the epistemic uncertainty, often to prompt insights about what could happen in the future, in an effort to develop new interventions. (C) Forecasting designs postulate multiple hypotheses in the parameter space, with an appropriately weighted average of outcomes constituting a well-calibrated forecast, given the current uncertainty. The shaded regions represent the current confidence intervals for the uncertainty parameters.

Figure 3.

Overview of scenario design process. First, determine the purpose of the scenario modeling exercise, including the questions to be addressed and the intended audience. This purpose informs all other design decisions. The taxonomies defined in Figures 1 and 2 should be applied at this step. Then, define the features that distinguish scenarios and those that are common across scenarios. Last, consider other design issues that may be relevant during all phases of scenario design.