Methods for Communicating the Impact of Parameter Uncertainty in a Multiple-Strategies Cost-Effectiveness Comparison

Abstract

Purpose: Analyzing and communicating uncertainty is essential in medical decision making. To judge whether risks are acceptable, policy makers require information on the expected outcomes but also on the uncertainty and potential losses related to the chosen strategy. We aimed to compare methods used to represent the impact of uncertainty in decision problems involving many strategies, enhance existing methods, and provide an open-source and easy-to-use tool.

Methods: We conducted a systematic literature search to identify methods used to represent the impact of uncertainty in cost-effectiveness analyses comparing multiple strategies. We applied the identified methods to probabilistic sensitivity analysis outputs of 3 published decision-analytic models comparing multiple strategies. Subsequently, we compared the following characteristics: type of information conveyed, use of a fixed or flexible willingness-to-pay threshold, output interpretability, and the graphical discriminatory ability. We further proposed adjustments and integration of methods to overcome identified limitations of existing methods.

Results: The literature search resulted in the selection of 9 methods. The 3 methods with the most favorable characteristics to compare many strategies were 1) the cost-effectiveness acceptability curve (CEAC) and cost-effectiveness acceptability frontier (CEAF), 2) the expected loss curve (ELC), and 3) the incremental benefit curve (IBC). The information required to assess confidence in a decision often includes the average loss and the probability of cost-effectiveness associated with each strategy. Therefore, we proposed the integration of information presented in an ELC and CEAC into a single heat map.

Conclusions: This article presents an overview of methods presenting uncertainty in multiple-strategy cost-effectiveness analyses, with their strengths and shortcomings. We proposed a heat map as an alternative method that integrates all relevant information required for health policy and medical decision making.

Highlights: To assess confidence in a chosen course of action, decision makers require information on both the probability and the consequences of making a wrong decision.This article contains an overview of methods for presenting uncertainty in multiple-strategy cost-effectiveness analyses.We propose a heat map that combines the probability of cost-effectiveness from the cost-effectiveness acceptability curve (CEAC) with the consequences of a wrong decision from the expected loss curve.Collapsing of the CEAC can be reduced by relaxing the CEAC, as proposed in this article.Code in Microsoft Excel and R is provided to easily analyze data using the methods discussed in this article.

Keywords: CEA; CEAC; ELC; PSA; many strategies; sensitivity analysis.

Figure 1.

Flow chart of the systematic literature search. The first search round used forward snowballing to identify the titles of articles that referenced the original articles introducing the methodology of the cost-effectiveness acceptability curve (CEAC) and cost-effectiveness acceptability frontier (CEAF). The second search round used backward snowballing to identify articles referenced by articles selected in the first search round. In both the first and second search rounds, the selection criteria were applied to titles, abstracts, and full papers. Nine methods were identified from the 24 selected articles.

Figure 2

Illustrative comparison of methods to communicate the impact of uncertainty with a willingness-to-pay axis. (A) The cost-effectiveness acceptability curve (CEAC) and its frontier (dashed black line).^, (B) The expected loss curves (ELCs) and their frontiers (dashed black line). (C) The expected benefit plot with upper and lower limit of the 95% prediction interval (dashed lines). All 3 methods use the x-axis to depict a range of willingness-to-pay threshold values, whereas the y-axis is used to show probabilities of cost-effectiveness for the CEAC, expected loss values for the ELC, and net monetary benefit for the expected benefit plot. The frontiers show which strategies have the highest expected net monetary benefit.

Figure 3

Illustrative comparison of methods that communicate the impact of uncertainty with a fixed willingness-to-pay threshold. The methods are (A) net benefit density plot, (B) stochastic dominance plot, (C) incremental benefit density plot, (D) incremental benefit curve, (E) return-risk space, and (F) cumulative rankogram. To produce these plots, the willingness-to-pay threshold was fixed at 50,000 €/quality-adjusted life-year for all figures. The probability density plots are normalized smoothened histograms of the net monetary benefits, using 100 and 500 bins for A and C, and the smoothening parameter was set at 0.5 (see Appendix 1C,D for the smoothening algorithm).

Figure 4

Adaptations of the cost-effectiveness acceptability curve (CEAC) and expected loss curve (ELC): the heat map and relaxed CEAC. The heat map in (A) shows the CEAC with expected loss values on the color scale (red representing high loss and blue low loss), and the heat map in (B) shows the ELC with the probability of being cost-effective on the color scale (red representing high probabilities and blue low probabilities). In (C), the graphical discriminatory ability of the CEAC is improved by relaxation of the CEAC. In the heat map in (D), graphical discriminatory ability of the color scale is improved by using the probability of being cost-effective of the relaxed CEAC on the ELC. The short vertical lines on the x-axis correspond to the incremental cost-effectiveness ratios of the strategies on the cost-effectiveness acceptability frontier, and the numbers denote which strategies are cost-effective in each interval of willingness-to-pay values. Strategies with a net monetary benefit (NMB) ≥99.95% of the maximum NMB value were considered cost-effective in the relaxed CEAC used for Figure 4C,D.