Predicting COVID-19 prognosis in the ICU remained challenging: external validation in a multinational regional cohort

Abstract

Objectives: Many prediction models for coronavirus disease 2019 (COVID-19) have been developed. External validation is mandatory before implementation in the intensive care unit (ICU). We selected and validated prognostic models in the Euregio Intensive Care COVID (EICC) cohort.

Study design and setting: In this multinational cohort study, routine data from COVID-19 patients admitted to ICUs within the Euregio Meuse-Rhine were collected from March to August 2020. COVID-19 models were selected based on model type, predictors, outcomes, and reporting. Furthermore, general ICU scores were assessed. Discrimination was assessed by area under the receiver operating characteristic curves (AUCs) and calibration by calibration-in-the-large and calibration plots. A random-effects meta-analysis was used to pool results.

Results: 551 patients were admitted. Mean age was 65.4 ± 11.2 years, 29% were female, and ICU mortality was 36%. Nine out of 238 published models were externally validated. Pooled AUCs were between 0.53 and 0.70 and calibration-in-the-large between -9% and 6%. Calibration plots showed generally poor but, for the 4C Mortality score and Spanish Society of Infectious Diseases and Clinical Microbiology (SEIMC) score, moderate calibration.

Conclusion: Of the nine prognostic models that were externally validated in the EICC cohort, only two showed reasonable discrimination and moderate calibration. For future pandemics, better models based on routine data are needed to support admission decision-making.

Keywords: COVID-19; Critical care; Intensive care unit; Prediction; Prognosis; SARS-CoV-2.

Fig. 1

Flowchart identifying prediction models. COVID-19, coronavirus disease 2019; ICU, intensive care unit; ARDS, acute respiratory distress syndrome; ASAT, aspartate aminotransferase. Legend: models for diagnosis and identifying people at risk in the general population were excluded. The remaining models were mainly prognostic, and further selection was based on outcome measures. As our cohort was composed of ICU patients only, in whom severe COVID-19 infection can be assumed, the outcome ICU admission, as well as progression to severe COVID-19, severe COVID-19, and ARDS, were excluded. Outcome measures length of hospital stay, in-hospital mortality, and in-hospital or out-of-hospital mortality were used. Since reporting of predictors and coefficients are necessary in order to validate prediction models as specifically assessed in step 4.9 in PROBAST [15], a tool to assess the risk of bias and applicability of prediction model studies, models which did not report or probably did not report this, or were machine learning or artificial intelligence studies, were excluded. Finally, predictors included in one of the final 21 prediction models were evaluated. Again, as we only included ICU patients and our goal was to validate models containing routinely available data, models including symptoms not relevant for ICU patients, not routinely available data, or data that were not available in the EICC cohort (e.g., ≥50% missing data) were excluded. Additionally, two promising models, which were not available in the COVID-PRECISE, were added. Abbreviations: PROBAST, Prediction model study Risk Of Bias Assessment Tool; EICC, Euregio Intensive Care COVID; COVID-PRECISE, Precise Risk Estimation to optimise COVID-19 Care for Infected or Suspected patients in diverse settings.

Fig. 2

Flowchart Euregio Intensive Care COVID cohort [16].

Fig. 3

Calibration plots prediction models. The cohort was divided into deciles according to the estimated probability score, displayed by points in the calibration plot.