An Easy and Quick Risk-Stratified Early Forewarning Model for Septic Shock in the Intensive Care Unit: Development, Validation, and Interpretation Study

Abstract

Background: Septic shock (SS) is a syndrome with high mortality. Early forewarning and diagnosis of SS, which are critical in reducing mortality, are still challenging in clinical management.

Objective: We propose a simple and fast risk-stratified forewarning model for SS to help physicians recognize patients in time. Moreover, further insights can be gained from the application of the model to improve our understanding of SS.

Methods: A total of 5125 patients with sepsis from the Medical Information Mart for Intensive Care-IV (MIMIC-IV) database were divided into training, validation, and test sets. In addition, 2180 patients with sepsis from the eICU Collaborative Research Database (eICU) were used as an external validation set. We developed a simplified risk-stratified early forewarning model for SS based on the weight of evidence and logistic regression, which was compared with multi-feature complex models, and clinical characteristics among risk groups were evaluated.

Results: Using only vital signs and rapid arterial blood gas test features according to feature importance, we constructed the Septic Shock Risk Predictor (SORP), with an area under the curve (AUC) of 0.9458 in the test set, which is only slightly lower than that of the optimal multi-feature complex model (0.9651). A median forewarning time of 13 hours was calculated for SS patients. 4 distinct risk groups (high, medium, low, and ultralow) were identified by the SORP 6 hours before onset, and the incidence rates of SS in the 4 risk groups in the postonset interval were 88.6% (433/489), 34.5% (262/760), 2.5% (67/2707), and 0.3% (4/1301), respectively. The severity increased significantly with increasing risk in both clinical features and survival. Clustering analysis demonstrated a high similarity of pathophysiological characteristics between the high-risk patients without SS diagnosis (NS_HR) and the SS patients, while a significantly worse overall survival was shown in NS_HR patients. On further exploring the characteristics of the treatment and comorbidities of the NS_HR group, these patients demonstrated a significantly higher incidence of mean blood pressure <65 mmHg, significantly lower vasopressor use and infused volume, and more severe renal dysfunction. The above findings were further validated by multicenter eICU data.

Conclusions: The SORP demonstrated accurate forewarning and a reliable risk stratification capability. Among patients forewarned as high risk, similar pathophysiological phenotypes and high mortality were observed in both those subsequently diagnosed as having SS and those without such a diagnosis. NS_HR patients, overlooked by the Sepsis-3 definition, may provide further insights into the pathophysiological processes of SS onset and help to complement its diagnosis and precise management. The importance of precise fluid resuscitation management in SS patients with renal dysfunction is further highlighted. For convenience, an online service for the SORP has been provided.

Keywords: early forewarning; machine learning; risk stratification; septic shock.

Figure 1

The flowchart of this study. The study flowchart comprises 4 stages. (A) Model development: Electronic health record (EHR) data were collected from the Medical Information Mart for Intensive Care-IV (MIMIC-IV) and eICU Collaborative Research Database (eICU), and multiple forewarning models were constructed. Using Shapley Additive Explanations (SHAP) analysis, we identified key features within the same test batch, which were then used to build the Septic Shock Risk Predictor (SORP) model. The performance of the SORP was compared to that of models based on all available features to evaluate whether the SORP performance loss is acceptable. (B) Risk stratification of septic shock (SS): Patients were stratified into risk groups based on SORP scores. (C) Clinical characteristics of the risk groups: We analyzed clinical characteristics across risk groups, with a focused analysis on high-risk nonshock patients. (D) Model deployment: An online tool was developed for clinical use. HR: high risk; LR: low risk; MR: medium risk; ULR: ultralow risk; WOE: weight of evidence.

Figure 2

Construction of the Septic Shock Risk Predictor (SORP) and derivation of patient risk stratification. (A) Shapley Additive Explanations (SHAP) summary plot for the top 10 clinical features contributing to the XGBoost model. The position on the y-axis is determined by the feature, in which “hotpink” represents blood gases and “cyan” represents vital signs, and that on the x-axis is determined by the Shapley value. The color from blue to red represents the feature values from low to high. (B) Receiver operating characteristic curve and area under the curve (AUC) for SORP forewarning performance. (C) Kolmogorov-Smirnov (KS) curve shows the difference between the cumulative proportion of septic shock (SS) patients and the cumulative proportion of septic nonshock (NS) patients as the risk score increases (blue line). (D) Stability of SS distribution across datasets. (E) A fitted curve showing the monotonic relationship between the risk score and SS probability in the postonset interval, according to the locally estimated scatterplot smoothing (LOESS) method. HR: high risk; LR: low risk; MR: medium risk; PSI: population stability index; ULR: ultralow risk.

Figure 3

Characteristics of different risk groups in the postonset interval. (A) The clinical features are standardized such that all means are scaled to 0 and SDs to 1. A value of 1 for the standardized mean value (y-axis) signifies that the mean value for the risk group was 1 SD higher than the mean value for the 4 risk groups shown in the graph as a whole. (B) Boxplot showing the changing trend for each clinical feature with risk groups. The y-axis shows the standardized value for each clinical feature. (C-F) Kaplan-Meier curves and survival analysis. (C) Overall survival for all patients. (D) Overall survival for septic nonshock patients. (E) Overall survival for septic shock patients in different risk groups. (F) Time from sepsis to septic shock. HR: high risk; LR: low risk; MR: medium risk; ULR: ultralow risk.

Figure 4

Characteristics of septic nonshock high risk (NS_HR) patients. (A) Clustering heatmap showing the similarity among risk groups. Standardized values were used for clustering (ie, each feature is centered at the sample mean and scaled by its SD), and “Euclidean” and “average” were used for clustering distance and clustering method, respectively. The values in the box are the median raw values of the features. (B) Boxplots showing the distribution of clinical features across risk groups. Red indicates NS_HR patients, orange indicates septic shock (SS) patients, and green indicates septic nonshock (NS) patients. (C) The degree of variation in the features of the risk groups. The y-axis shows the standardized mean for each clinical feature. (D) Overall survival for SS, NS_HR, and NS_O (other risk groups of NS). (E) Bar plot of the polar coordinate showing the proportion of clinical events in each risk group of patients. The boxplot shows the values of kidney-related features during the look-back interval. We transformed the infusion volume (IV), used SS_LR’s IV as a reference (3815.25 mL), and expressed the IV in other risk groups as a percentage of that of SS_LR patients. CKD: the rate of patients with chronic kidney disease; HR: high risk; IV: the mean infusion volume of patients from the preonset interval to the postonset interval; LR: low risk; MBP <65: the rate of patients with mean blood pressure (MBP) <65 mmHg from the preonset interval to the postonset interval; MR: medium risk; ns: not significant (P>.05); ULR: ultralow risk; vaso: the rate of patients treated with vasopressors from the preonset interval to the postonset interval. *P≤.05, **P≤.01, ***P≤.001.

Figure 5

Characteristics of septic shock low risk (SS_LR) patients. (A) Mean value line and unbiased standard error bar for septic shock (SS) patients. Risk groups derived by the Septic Shock Risk Predictor (SORP) 6 hours before SS, and the change in risk over time for each group. The cyan dashed line represents 6 hours before the onset of SS, and the red dashed line represents the onset time of SS. (B) The rate of invasive procedures of each risk group in SS patients. P values are derived from SS_LR patients compared to other SS patients (details in Multimedia Appendix 10). (C) The rate of invasive procedures of each risk group in LR patients identified by the SORP. Here, events are measured from 12 hours before the prediction to the time of the prediction. P values are derived from SS_LR patients compared to other septic nonshock low risk (NS_LR) patients (details in Multimedia Appendix 11). CRRT: continuous renal replacement therapy; HR: high risk; LR: low risk; MR: medium risk; ns: not significant.

Figure 6

Reproducibility of findings in external independent eICU Collaborative Research Database (eICU) data. (A) The clinical features are standardized with all means scaled to 0 and SDs to 1. A value of 1 for the standardized mean value (y-axis) signifies that the mean value for the risk group was 1 SD higher than the mean value for the 4 risk groups shown in the graph. (B-D) Overall survival for all patients, septic nonshock (NS) patients, and septic shock (SS) patients. (E) Clustering heatmap showing the similarity among risk groups. Standardized values were used for clustering (ie, each feature is centered at the sample mean and scaled by its SD), and “Euclidean” and “average” were used for clustering distance and clustering method, respectively. Red indicates higher levels, and blue indicates lower levels. (F) Characteristics of risk groups. The y-axis shows the standardized mean for each clinical feature. (G) Overall survival for SS, NS_HR, and NS_O (other risk groups of NS). (H) The bar plot of the polar coordinate shows the proportion of clinical events in each risk group of patients. The starting and ending times of patient infusions are not provided in the eICU, and thus, it was not possible to estimate the infusion volume of patients within a specific interval. HR: high risk; LR: low risk; MBP <65: the rate of patients with mean blood pressure (MBP) <65 mmHg from the preonset interval to the postonset interval; MR: medium risk; ULR: ultralow risk; vaso: the rate of patients treated with vasopressors from the preonset interval to the postonset interval.