Dynamic Mortality Risk Predictions for Children in ICUs: Development and Validation of Machine Learning Models

Abstract

Objectives: Assess a machine learning method of serially updated mortality risk.

Design: Retrospective analysis of a national database (Health Facts; Cerner Corporation, Kansas City, MO).

Setting: Hospitals caring for children in ICUs.

Patients: A total of 27,354 admissions cared for in ICUs from 2009 to 2018.

Interventions: None.

Main outcome: Hospital mortality risk estimates determined at 6-hour time periods during care in the ICU. Models were truncated at 180 hours due to decreased sample size secondary to discharges and deaths.

Measurements and main results: The Criticality Index, based on physiology, therapy, and care intensity, was computed for each admission for each time period and calibrated to hospital mortality risk (Criticality Index-Mortality [CI-M]) at each of 29 time periods (initial assessment: 6 hr; last assessment: 180 hr). Performance metrics and clinical validity were determined from the held-out test sample (n = 3,453, 13%). Discrimination assessed with the area under the receiver operating characteristic curve was 0.852 (95% CI, 0.843-0.861) overall and greater than or equal to 0.80 for all individual time periods. Calibration assessed by the Hosmer-Lemeshow goodness-of-fit test showed good fit overall (p = 0.196) and was statistically not significant for 28 of the 29 time periods. Calibration plots for all models revealed the intercept ranged from--0.002 to 0.009, the slope ranged from 0.867 to 1.415, and the R2 ranged from 0.862 to 0.989. Clinical validity assessed using population trajectories and changes in the risk status of admissions (clinical volatility) revealed clinical trajectories consistent with clinical expectations and greater clinical volatility in deaths than survivors (p < 0.001).

Conclusions: Machine learning models incorporating physiology, therapy, and care intensity can track changes in hospital mortality risk during intensive care. The CI-M's framework and modeling method are potentially applicable to monitoring clinical improvement and deterioration in real time.

Figure 1.

Area under the receiver operating characteristic curves (AUROC) for all Criticality Index – Mortality models and for all models combined. a. Separate models were developed for 29 6-hour time periods from hour 6 to hour 180. The dots are AUROCs for the individual models and the shaded areas are the 95% confidence intervals b. Composite AUROC. All models have been combined.

Figure 2.

Calibration plot performance metrics for all Criticality Index – Mortality models and for all models combined a. Separate models were developed for 29 6-hour time periods from hour 6 to hour 180. Calibration plot metrics include the regression line intercepts (black), slopes (tan) and R²’s (blue). The shaded areas are the 95% confidence intervals. b. Composite calibration plot. The dashed line is the line of identity and the solid line is the regression line. The plot includes 294 risk intervals from all models. The comparison of observed proportions to expected proportions of outcome for each interval were not statistically different (p > 0.05) for 290 (98.64%) of the risk intervals. Each interval had ≥ 170 admissions. The Hosmer-Lemeshow goodness-of-fit test for the overall calibration was 0.195.

Figure 3.

Trajectories for high risk deaths and survivors and low risk deaths and survivors. Risk was computed with the Criticality Index – Mortality. High risk indicates the highest mortality risk decile determined from the initial time period. The shaded areas are the 95% confidence intervals. The mortality risks cutpoints for the high risk cohorts were 15.1% for the deaths and 11.3% for the survivors. The trajectories were constructed from the total sample with the following sample sizes: low risk survivors: 24,168, admissions, high risk survivors: 2,685 admissions, low risk deaths: 451 admissions, high risk deaths: 50 patients.

Figure 4.

Change in mortality risk for individual admissions in consecutive time periods. Risk was computed with the Criticality Index – Mortality. Data are shown for survivors (tan) and deaths (black). The frequencies (vertical axis) are standardized such that the added areas under the bars are 1 for each outcome group in each figure. a. Change in mortality risk for all consecutive time intervals. Deaths demonstrate increased clinical volatility compared to survivors. The average increase in mortality was 0.021 for deaths and 0.006 for survivors (p<0.001) and the average decrease was 0.022 for deaths and 0.008 for survivors (p<0.001). b. Maximum mortality risk increase (clinical deterioration). The average maximum deterioration for deaths was 0.050 and 0.015 for survivors (p<0.001) c. Maximum mortality risk decrease (clinical improvement). The average maximum decrease was 0.063 for deaths and 0.022 for survivors (p<0.001).