A generalizable and interpretable model for mortality risk stratification of sepsis patients in intensive care unit

Abstract

Purpose: This study aimed to construct a mortality model for the risk stratification of intensive care unit (ICU) patients with sepsis by applying a machine learning algorithm.

Methods: Adult patients who were diagnosed with sepsis during admission to ICU were extracted from MIMIC-III, MIMIC-IV, eICU, and Zigong databases. MIMIC-III was used for model development and internal validation. The other three databases were used for external validation. Our proposed model was developed based on the Extreme Gradient Boosting (XGBoost) algorithm. The generalizability, discrimination, and validation of our model were evaluated. The Shapley Additive Explanation values were used to interpret our model and analyze the contribution of individual features.

Results: A total of 16,741, 15,532, 22,617, and 1,198 sepsis patients were extracted from the MIMIC-III, MIMIC-IV, eICU, and Zigong databases, respectively. The proposed model had an area under the receiver operating characteristic curve (AUROC) of 0.84 in the internal validation, which outperformed all the traditional scoring systems. In the external validations, the AUROC was 0.87 in the MIMIC-IV database, better than all the traditional scoring systems; the AUROC was 0.83 in the eICU database, higher than the Simplified Acute Physiology Score III and Sequential Organ Failure Assessment (SOFA),equal to 0.83 of the Acute Physiology and Chronic Health Evaluation IV (APACHE-IV), and the AUROC was 0.68 in the Zigong database, higher than those from the systemic inflammatory response syndrome and SOFA. Furthermore, the proposed model showed the best discriminatory and calibrated capabilities and had the best net benefit in each validation.

Conclusions: The proposed algorithm based on XGBoost and SHAP-value feature selection had high performance in predicting the mortality of sepsis patients within 24 h of ICU admission.

Keywords: In-ICU mortality; Multi-source data; Risk stratification; SHAP; Sepsis; XGBoost.

Fig. 1

Flowchart of data processing and model development

Fig. 2

SHAP summary plot of the proposed model

Fig. 3

SHAP dependency plots of the top 15 features. The X-axis represents the actual value of the feature, the Y-axis represents the SHAP value of the feature, and the points correspond to the samples in the training set. A SHAP value above zero indicates an increased risk of in-ICU mortality

Fig. 4

Performance evaluation of different ML methods

Fig. 5

Non-nested and nested cross-validation on XGBoost in training dataset

Fig. 6

The area under the receiver operating characteristic (AUROC) curve. Internal validation performance: A The new prediction model in the MIMIC-III population during ICU hospitalization (iii_hosp) versus the OASIS, APACH III, SAPS II, LODS, SIRS, and SOFA models. External validation performance: B The new model used in the MIMIC-IV population (iv_hosp) versus the OASIS, APACH III, SAPS II, LODS, SIRS, and SOFA systems. C The new model used in the eICU population (eicu_hosp) versus the SAPS III, APACH IV, and SOFA scores. D The new model used in the Zigong population (Zigong_hosp) versus the SOFA and SIRS scores

Fig. 7

Decision curve analysis of proposed model and traditional scoring systems in four populations: A MIMIC-III population; B MIMIC-IV population; C eICU population; D Zigong population

Fig. 8

Calibration and discrimination potentials of the proposed model and traditional scoring systems in external validation. A–G The external validation results in the MIMIC-IV dataset: A The new model (Brier score = 0.068; C-index = 0.862; discrimination slope = 0.303); B OASIS (Brier score = 0.097; C-index = 0.795; discrimination slope = 0.215); C APS III (Brier score = 0.079; C-index = 0.857; discrimination slope = 0.287); D SAPS II score (Brier score = 0.126; C-index = 0.794; discrimination slope = 0.275); E LODS score (Brier score = 0.082; C-index = 0.844; discrimination slope = 0.222); F SOFA score (Brier score = 0.0821; C-index = 0.773; discrimination slope = 0.123); G SIRS score (Brier score = 0.097; C-index = 0.621; discrimination slope = 0.011). H–K The external validation results in the eICU dataset: H The new model (Brier score = 0.083; C-index = 0.820; discrimination slope = 0.270); I SAPS III (Brier score = 0.081; C-index = 0.782; discrimination slope = 0.154); J APACHE IV score (Brier score = 0.091; C-index = 0.826; discrimination slope = 0.290); K SOFA score (Brier score = 0.090; C-index = 0.714; discrimination slope = 0.060). L–N The external validation results in the Zigong dataset: L The new model (Brier score = 0.210; C-index = 0.679; discrimination slope = 0.123); M SOFA score (Brier score = 0.250; C-index = 0.505; discrimination slope = 0.003); N SIRS score (Brier score = 0.285; C-index = 0.469; discrimination slope = -0.002)