Assessing the Clinical Impact of Risk Prediction Models With Decision Curves: Guidance for Correct Interpretation and Appropriate Use

Abstract

The decision curve is a graphical summary recently proposed for assessing the potential clinical impact of risk prediction biomarkers or risk models for recommending treatment or intervention. It was applied recently in an article in Journal of Clinical Oncology to measure the impact of using a genomic risk model for deciding on adjuvant radiation therapy for prostate cancer treated with radical prostatectomy. We illustrate the use of decision curves for evaluating clinical- and biomarker-based models for predicting a man's risk of prostate cancer, which could be used to guide the decision to biopsy. Decision curves are grounded in a decision-theoretical framework that accounts for both the benefits of intervention and the costs of intervention to a patient who cannot benefit. Decision curves are thus an improvement over purely mathematical measures of performance such as the area under the receiver operating characteristic curve. However, there are challenges in using and interpreting decision curves appropriately. We caution that decision curves cannot be used to identify the optimal risk threshold for recommending intervention. We discuss the use of decision curves for miscalibrated risk models. Finally, we emphasize that a decision curve shows the performance of a risk model in a population in which every patient has the same expected benefit and cost of intervention. If every patient has a personal benefit and cost, then the curves are not useful. If subpopulations have different benefits and costs, subpopulation-specific decision curves should be used. As a companion to this article, we released an R software package called DecisionCurve for making decision curves and related graphics.

Fig 1.

Behavior of decision curves as a function of outcome prevalence. (A) The distribution of a biomarker X in cases and controls. Decision curves for risks based on biomarker X are shown for prevalence (B) 10%, (C) 35%, and (D) 1%.

Fig 2.

The performance of risk models in a population (POP), which comprises two subpopulations S1 and S2. (A) Solid curves give the decision curves for risk models I and II in POP. Dashed and dotted lines give the decision curves for model II in subpopulations S1 and S2; model I has the same performance in both subpopulations. The population decision curves show that the two risk models are similar in the population. However, for subpopulation S2, the subpopulation net benefit (NB) is higher for risk model II (dotted line). For subpopulation S1, NB is higher for risk model I (dashed line). (B) For risk model I, the distribution of predicted risks is the same for S1 and S2. Therefore, if S1 and S2 have different risk thresholds, the subpopulation NB for risk model I is the same as the NB in POP. (C) For risk model II, the distributions of predicted risks differ for S1 and S2. This explains the divergence of the population and subpopulation-specific decision curves for risk model II in (A).

Fig 3.

(A) Decision curves for two risk models for prostate cancer produced with DecisionCurve software. The vertical axis displays standardized net benefit. The two horizontal axes show the correspondence between risk threshold and cost:benefit ratio. (B) Clinical impact curve for the biomarker-based risk model. Of 1,000 patients, the heavy blue solid line shows the total number who would be deemed high risk for each risk threshold. The gold dashed line shows how many of those would be true positives (cases). (C) True- and false-positive rates as functions of the risk threshold, for the biomarker-based risk model. The figure shows information similar to that of a receiver operating characteristic curve and also shows the risk threshold corresponding to each true- and false-positive rate. Bands on all plots represent pointwise 95% CIs constructed via bootstrapping.