Making Drug Approval Decisions in the Face of Uncertainty: Cumulative Evidence versus Value of Information

Abstract

Background: The COVID-19 pandemic underscored the criticality and complexity of decision making for novel treatment approval and further research. Our study aims to assess potential decision-making methodologies, an evaluation vital for refining future public health crisis responses.

Methods: We compared 4 decision-making approaches to drug approval and research: the Food and Drug Administration's policy decisions, cumulative meta-analysis, a prospective value-of-information (VOI) approach (using information available at the time of decision), and a reference standard (retrospective VOI analysis using information available in hindsight). Possible decisions were to reject, accept, provide emergency use authorization, or allow access to new therapies only in research settings. We used monoclonal antibodies provided to hospitalized COVID-19 patients as a case study, examining the evidence from September 2020 to December 2021 and focusing on each method's capacity to optimize health outcomes and resource allocation.

Results: Our findings indicate a notable discrepancy between policy decisions and the reference standard retrospective VOI approach with expected losses up to $269 billion USD, suggesting suboptimal resource use during the wait for emergency use authorization. Relying solely on cumulative meta-analysis for decision making results in the largest expected loss, while the policy approach showed a loss up to $16 billion and the prospective VOI approach presented the least loss (up to $2 billion).

Conclusion: Our research suggests that incorporating VOI analysis may be particularly useful for research prioritization and treatment implementation decisions during pandemics. While the prospective VOI approach was favored in this case study, further studies should validate the ideal decision-making method across various contexts. This study's findings not only enhance our understanding of decision-making strategies during a health crisis but also provide a potential framework for future pandemic responses.

Highlights: This study reviews discrepancies between a reference standard (retrospective VOI, using hindsight information) and 3 conceivable real-time approaches to research-treatment decisions during a pandemic, suggesting suboptimal use of resources.Of all prospective decision-making approaches considered, VOI closely mirrored the reference standard, yielding the least expected value loss across our study timeline.This study illustrates the possible benefit of VOI results and the need for evidence accumulation accompanied by modeling in health technology assessment for emerging therapies.

Keywords: COVID-19; antibodies; cost-benefit analysis; decision support techniques; drug approval; meta-analysis; monoclonal; policy analyses.

Figure 1

Decision matrix. This 2 × 2 figure shows the 4 potential combined research-treatment strategies, their advantages and disadvantages, the decision rule when this strategy would be selected as the optimal strategy, and how the expected net monetary benefit (ENB) would be calculated for the value-of-information (VOI) analyses to determine the optimal strategy according to the VOI. Difference retrospective/prospective VOI: the number of patients is based on the projections on the date of evaluation (prospective) or on the last data set on the timeline (retrospective). iNB, incremental net (monetary) benefit (NB treat – NB control); p, proportion of patients n in a new randomized controlled trial (RCT) randomly assigned to treatment; 1 − p, proportion of patients n in a new RCT randomly assigned to control; EVSI, expected value of sample information, calculated in comparison with the optimal treatment (i.e., over and above the iNB gained if iNB > 0); EVSI(n), EVSI dependent on sample size n of the new RCT; cost RCT(n), fixed cost + variable cost of performing a new RCT dependent on sample size n. The optimal sample size is determined by maximizing the function in the quadrant with respect to n. Implementation and reversal costs are assumed to be 0 given that the intervention concerns a guideline change.

Figure 2

Forest plot of the cumulative meta-analysis. Pooled relative risk for mortality with treatment with monoclonal antibodies versus control arms. At each time point the trial results were published: the relative risk is a pooled result of the newly included study and the evidence thus far accrued. K, number of included studies; Sari, sarilumab; TCZ/Toci, tocilizumab.

Figure 3

Incremental cost-effectiveness planes of the monoclonal antibody (MAb) treatment versus usual care over time. The x-axes show incremental effectiveness, whereas the y-axes show incremental costs in USD. This grid slot shows the cost-effectiveness (CE) plot at each time point at which a new study (k) is added to the cumulative meta-analysis treatment effectiveness estimate. The plots are based on the results of 10,000 iterations. The yellow circle depicts the estimate for quality-adjusted life-years (QALYs) and the gray for LY. The dotted line surrounding the mean estimate reflects the uncertainty. Other parameter inputs remained constant over time.

Figure 4

Timeline of publications and 4 approaches to decision making. The plot represents the publication of the results of each study and the optimal strategy as suggested by the 4 approaches (cumulative meta-analysis, policy set by the Food and Drug Administration, prospective value of information [VOI] and retrospective VOI).

Figure 5

Expected value loss from choosing the suboptimal strategy according to the gold standard compared with the other approaches. This is calculated by taking the EV of the optimal strategy—the EV of the strategy suggested by each respective approach (policy, CMA, or prospective VOI). The EV is provided by the gold standard, as this approach is closest to knowing the true EV as the largest proportion of hospitalizations have already been observed. The smaller figure in the top right corner is the same plot with a zoomed-in version of the plot with a y-axis set to max $25,000 million. The highest losses for each strategy were CMA approach: $269 billion; policy approach: $16 billion; prospective VOI approach: $2 billion. CMA, cumulative meta-analysis; EV, expected value; VOI, value of information.