The Impact of Multi-Institution Datasets on the Generalizability of Machine Learning Prediction Models in the ICU

Abstract

Objectives: To evaluate the transferability of deep learning (DL) models for the early detection of adverse events to previously unseen hospitals.

Design: Retrospective observational cohort study utilizing harmonized intensive care data from four public datasets.

Setting: ICUs across Europe and the United States.

Patients: Adult patients admitted to the ICU for at least 6 hours who had good data quality.

Interventions: None.

Measurements and main results: Using carefully harmonized data from a total of 334,812 ICU stays, we systematically assessed the transferability of DL models for three common adverse events: death, acute kidney injury (AKI), and sepsis. We tested whether using more than one data source and/or algorithmically optimizing for generalizability during training improves model performance at new hospitals. We found that models achieved high area under the receiver operating characteristic (AUROC) for mortality (0.838-0.869), AKI (0.823-0.866), and sepsis (0.749-0.824) at the training hospital. As expected, AUROC dropped when models were applied at other hospitals, sometimes by as much as -0.200. Using more than one dataset for training mitigated the performance drop, with multicenter models performing roughly on par with the best single-center model. Dedicated methods promoting generalizability did not noticeably improve performance in our experiments.

Conclusions: Our results emphasize the importance of diverse training data for DL-based risk prediction. They suggest that as data from more hospitals become available for training, models may become increasingly generalizable. Even so, good performance at a new hospital still depended on the inclusion of compatible hospitals during training.

Figure 1.

Schematic overview of the experimental setup. CORAL = correlation alignment.

Figure 2.

Performance of the ICU mortality prediction model when trained on one dataset (rows) and evaluated on others (columns). A, Area under the receiver operating characteristic (AUROC). B, Area under the precision-recall curve (AUPRC). C, Sensitivity. D, Positive predictive value (PPV). The diagonal represents internal validation, that is, training and test samples were taken from the same dataset. In pooled (n–1), the model was trained on combined data from all except the test dataset. In all, training data from all datasets (including the test dataset) was used during model development. Sensitivity and PPV were evaluated on the stay level and thresholds were chosen based on the validation portion of the training dataset. Performance was averaged across five random resamples. All models used a gated recurrent unit featurizer. AUMCdb = Amsterdam University Medical Centers database, eICU = eICU Collaborative Research Database, HiRID = High Time Resolution ICU Dataset, MIMIC = Medical Information Mart for Intensive Care IV.

Figure 3.

Performance of the acute kidney injury prediction model. A, Area under the receiver operating characteristic (AUROC). B, Area under the precision-recall curve (AUPRC). C, Sensitivity. D, Positive predictive value (PPV). All models used a gated recurrent unit featurizer. AUMCdb = Amsterdam University Medical Centers database, eICU = eICU Collaborative Research Database, HiRID = High Time Resolution ICU Dataset, MIMIC = Medical Information Mart for Intensive Care IV.

Figure 4.

Performance of the sepsis prediction model. A, Area under the receiver operating characteristic (AUROC). B, Area under the precision-recall curve (AUPRC). C, Sensitivity. D, Positive predictive value (PPV). All models used a gated recurrent unit featurizer. AUMCdb = Amsterdam University Medical Centers database, eICU = eICU Collaborative Research Database, HiRID = High Time Resolution ICU Dataset, MIMIC = Medical Information Mart for Intensive Care IV.