Prediction Model for 30-Day Mortality after Non-Cardiac Surgery Using Machine-Learning Techniques Based on Preoperative Evaluation of Electronic Medical Records

Abstract

Background: Machine-learning techniques are useful for creating prediction models in clinical practice. This study aimed to construct a prediction model of postoperative 30-day mortality based on an automatically extracted electronic preoperative evaluation sheet.

Methods: We used data from 276,341 consecutive adult patients who underwent non-cardiac surgery between January 2011 and December 2020 at a tertiary center for model development and internal validation, and another dataset from 63,384 patients between January 2011 and October 2021 at another center for external validation. Postoperative 30-day mortality was 0.16%. We developed an extreme gradient boosting (XGB) prediction model using only variables from preoperative evaluation sheets.

Results: The model yielded an area under the curve of 0.960 and an area under the precision and recall curve of 0.216, which were 0.932 and 0.122, respectively, in the external validation set. The optimal threshold calculated by Youden's J statistic had a sensitivity of 0.885 and specificity of 0.914. In an additional analysis with balanced distribution, the model showed a similar predictive value.

Conclusion: We presented a machine-learning prediction model for 30-day mortality after non-cardiac surgery using preoperative variables automatically extracted from electronic medical records and validated the model in a multi-center setting. Our model may help clinicians predict postoperative outcomes.

Keywords: artificial intelligence; machine learning; mortality; prognosis; risk; surgery.

Figure 1

Overview of study. AUROC denotes the area under the receiver operating characteristic curve). AUPRC indicates the area under the precision and recall curve.

Figure 2

AUROC (area under the receiver operating characteristic curve) and AUPRC (area under the precision and recall curve) plots in the internal validation test set (Samsung medical center) and external validation dataset (Ajou university medical center). XGB (extreme gradient boosting algorithm) denotes the main model of our study based on various variables from preoperative evaluation sheets. Baseline denotes logistic regression model with only sex, age, weight, height, American Society of Anesthesiologists (ASA) class, and underlying disease.

Figure 3

Calibration plot between model-predicted probability and true probability.

Figure 4

SHapley Additive exPlanations (SHAP) bap plot shows the importance of features in the prediction model by calculating the mean absolute SHAP value for each feature.

Figure 5

SHapley Additive exPlanations (SHAP) beeswarm plot shows a summary of how the top features in a dataset impact the model’s output. A red color means a high feature value. The blue color means a low feature value. Each point represents an individual person. The horizontal position of each point shows the impact of the feature on the model’s prediction. For example, in the case of ASA (American Society of Anesthesiologists) class 1, a high feature value (the red color) influences the model to predict less death. Conversely, in the case of age, the high feature value affects the model to predict more death.