How to interpret a small increase in AUC with an additional risk prediction marker: decision analysis comes through

Abstract

An important question in the evaluation of an additional risk prediction marker is how to interpret a small increase in the area under the receiver operating characteristic curve (AUC). Many researchers believe that a change in AUC is a poor metric because it increases only slightly with the addition of a marker with a large odds ratio. Because it is not possible on purely statistical grounds to choose between the odds ratio and AUC, we invoke decision analysis, which incorporates costs and benefits. For example, a timely estimate of the risk of later non-elective operative delivery can help a woman in labor decide if she wants an early elective cesarean section to avoid greater complications from possible later non-elective operative delivery. A basic risk prediction model for later non-elective operative delivery involves only antepartum markers. Because adding intrapartum markers to this risk prediction model increases AUC by 0.02, we questioned whether this small improvement is worthwhile. A key decision-analytic quantity is the risk threshold, here the risk of later non-elective operative delivery at which a patient would be indifferent between an early elective cesarean section and usual care. For a range of risk thresholds, we found that an increase in the net benefit of risk prediction requires collecting intrapartum marker data on 68 to 124 women for every correct prediction of later non-elective operative delivery. Because data collection is non-invasive, this test tradeoff of 68 to 124 is clinically acceptable, indicating the value of adding intrapartum markers to the risk prediction model.

Keywords: AUC; cesarean section; decision curves; receiver operating characteristic curves; relative utility curves.

Figure 1

ROC curves for risk prediction involving later non-elective operative delivery.

Figure 2

Relative utility (RU) curves for risk prediction involving later non-elective operative delivery. Vertical lines indicate risk thresholds

Figure 3

Example showing construction of a concave envelope of ROC points

Figure 4

Calibration plot involving risk prediction for later non-elective operative delivery. The dashed line is the 45 degree line reference for no bias. The arrow shows an example of converting a risk score to an estimated risk via interpolation.