Decision curve analysis confirms higher clinical utility of multi-domain versus single-domain prediction models in patients with open abdomen treatment for peritonitis

Abstract

Background: Prediction modelling increasingly becomes an important risk assessment tool in perioperative systems approaches, e.g. in complex patients with open abdomen treatment for peritonitis. In this population, combining predictors from multiple medical domains (i.e. demographical, physiological and surgical variables) outperforms the prediction capabilities of single-domain prediction models. However, the benefit of these prediction models for clinical decision-making remains to be investigated. We therefore examined the clinical utility of mortality prediction models in patients suffering from peritonitis with a decision curve analysis.

Methods: In this secondary analysis of a large dataset, a traditional logistic regression approach, three machine learning methods and a stacked ensemble were employed to examine the predictive capability of demographic, physiological and surgical variables in predicting mortality under open abdomen treatment for peritonitis. Calibration was examined with calibration belts and predictive performance was assessed with the area both under the receiver operating characteristic curve (AUROC) and under the precision recall curve (AUPRC) and with the Brier Score. Clinical utility of the prediction models was examined by means of a decision curve analysis (DCA) within a treatment threshold range of interest of 0-30%, where threshold probabilities are traditionally defined as the minimum probability of disease at which further intervention would be warranted.

Results: Machine learning methods supported available evidence of a higher prediction performance of a multi- versus single-domain prediction models. Interestingly, their prediction performance was similar to a logistic regression model. The DCA demonstrated that the overall net benefit is largest for a multi-domain prediction model and that this benefit is larger compared to the default "treat all" strategy only for treatment threshold probabilities above about 10%. Importantly, the net benefit for low threshold probabilities is dominated by physiological predictors: surgical and demographics predictors provide only secondary decision-analytic benefit.

Conclusions: DCA provides a valuable tool to compare single-domain and multi-domain prediction models and demonstrates overall higher decision-analytic value of the latter. Importantly, DCA provides a means to clinically differentiate the risks associated with each of these domains in more depth than with traditional performance metrics and highlighted the importance of physiological predictors for conservative intervention strategies for low treatment thresholds. Further, machine learning methods did not add significant benefit either in prediction performance or decision-analytic utility compared to logistic regression in these data.

Keywords: Calibration; Clinical prediction modelling; Decision curve analysis; Machine learning; Peritonitis.

Fig. 1

Model building and evaluation approach in this study. A detailed description of the approach is provided in the Methods section

Fig. 2

Calibration curves for the single-domain prediction models and multi-domain predictions models stratified according to the modelling approach. Shaded areas denote the 95%-confidence intervals. P-values regarding the quality of the calibration [10] and Brier-scores are shown for each prediction model and are summarized by the mean and 95%-confidence intervals. Black dashed lines indicate the Generalized Additive Model (GAM)-smoothed calibration belts from the ensemble of 100 individual calibration belts

Fig. 3

Decision curve analysis of single-domain and multi-domain models. The four domain-specific prediction models are compared to the two default strategies “Treat All” and “Treat None”. Note that the “Treat All” option crosses the zero benefit line at the prevalence of negative outcomes in our cohort and complete-case analysis (26.7%)

Fig. 4

Calibration (A) and decision curve analysis (B) for a stacked ensemble prediction model based on a multivariable logistic regression, an Elastic Net, a Random Forest and a Gradient Boosting Machine as base learners. The stacked ensemble is based on a Gradient Boosting Machine that predicts the mortality outcome based on the cross-validated predictions of the base learners. For calibration, shaded areas denote the 95%-confidence range and black dashed lines indicate the Generalized Additive Model (GAM)-smoothed calibration belts from the ensemble of 100 individual calibration belts. P-values regarding the quality of the calibration and Brier-scores are shown