Dynamic prediction of mortality in COVID-19 patients in the intensive care unit: A retrospective multi-center cohort study

Abstract

Background: The COVID-19 pandemic continues to overwhelm intensive care units (ICUs) worldwide, and improved prediction of mortality among COVID-19 patients could assist decision making in the ICU setting. In this work, we report on the development and validation of a dynamic mortality model specifically for critically ill COVID-19 patients and discuss its potential utility in the ICU.

Methods: We collected electronic medical record (EMR) data from 3222 ICU admissions with a COVID-19 infection from 25 different ICUs in the Netherlands. We extracted daily observations of each patient and fitted both a linear (logistic regression) and non-linear (random forest) model to predict mortality within 24 h from the moment of prediction. Isotonic regression was used to re-calibrate the predictions of the fitted models. We evaluated the models in a leave-one-ICU-out (LOIO) cross-validation procedure.

Results: The logistic regression and random forest model yielded an area under the receiver operating characteristic curve of 0.87 [0.85; 0.88] and 0.86 [0.84; 0.88], respectively. The recalibrated model predictions showed a calibration intercept of -0.04 [-0.12; 0.04] and slope of 0.90 [0.85; 0.95] for logistic regression model and a calibration intercept of -0.19 [-0.27; -0.10] and slope of 0.89 [0.84; 0.94] for the random forest model.

Discussion: We presented a model for dynamic mortality prediction, specifically for critically ill COVID-19 patients, which predicts near-term mortality rather than in-ICU mortality. The potential clinical utility of dynamic mortality models such as benchmarking, improving resource allocation and informing family members, as well as the development of models with more causal structure, should be topics for future research.

Fig. 1

Areas under the receiver operating characteristic curve (AUROCs) yielded by the logistic regression (LR) and random forest (RF) models in the different ICUs.

Fig. 2

Cumulative distributions for F iO2 of the samples taken within 24 h before death (i.e. ‘event samples’) of patients from ICU O(N = 31), P(N = 13), R(N = 16) and X(N = 16). The cumulative distribution of event samples of patients from all ICUs (N = 667) is plotted as references. Distributions were found significantly different (P < 0.05) from the reference based on a two-sided Kolmogorov-Smirnov (KS) test in ICU O (KS-statistic = 0.32, P = 0.011), P (KS-statistic = 0.43, P = 0.012), R (KS-statistic = 0.46, P = 0.002) and X (KS-statistic = 0.33, P = 0.046).

Fig. 3

Smoothed flexible calibration curves for (a) the logistic regression (LR) and (b) the random forest (RF) models, with and without re-calibration using isotonic regression. Shaded areas around the curves represent the 95%CIs. In the bottom plots, histograms of the predictions are shown.

Fig. 4

Summary plots for the SHAP values constructed from both logistic regression (a) and random forest model (b). Each SHAP value is represented by a single dot on each predictor row. Color is used to display the corresponding value of the predictor. Predictors are ordered by the mean SHAP magnitude. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)