Development and external validation of a prediction risk model for short-term mortality among hospitalized U.S. COVID-19 patients: A proposal for the COVID-AID risk tool

Abstract

Background: The 2019 novel coronavirus disease (COVID-19) has created unprecedented medical challenges. There remains a need for validated risk prediction models to assess short-term mortality risk among hospitalized patients with COVID-19. The objective of this study was to develop and validate a 7-day and 14-day mortality risk prediction model for patients hospitalized with COVID-19.

Methods: We performed a multicenter retrospective cohort study with a separate multicenter cohort for external validation using two hospitals in New York, NY, and 9 hospitals in Massachusetts, respectively. A total of 664 patients in NY and 265 patients with COVID-19 in Massachusetts, hospitalized from March to April 2020.

Results: We developed a risk model consisting of patient age, hypoxia severity, mean arterial pressure and presence of kidney dysfunction at hospital presentation. Multivariable regression model was based on risk factors selected from univariable and Chi-squared automatic interaction detection analyses. Validation was by receiver operating characteristic curve (discrimination) and Hosmer-Lemeshow goodness of fit (GOF) test (calibration). In internal cross-validation, prediction of 7-day mortality had an AUC of 0.86 (95%CI 0.74-0.98; GOF p = 0.744); while 14-day had an AUC of 0.83 (95%CI 0.69-0.97; GOF p = 0.588). External validation was achieved using 265 patients from an outside cohort and confirmed 7- and 14-day mortality prediction performance with an AUC of 0.85 (95%CI 0.78-0.92; GOF p = 0.340) and 0.83 (95%CI 0.76-0.89; GOF p = 0.471) respectively, along with excellent calibration. Retrospective data collection, short follow-up time, and development in COVID-19 epicenter may limit model generalizability.

Conclusions: The COVID-AID risk tool is a well-calibrated model that demonstrates accuracy in the prediction of both 7-day and 14-day mortality risk among patients hospitalized with COVID-19. This prediction score could assist with resource utilization, patient and caregiver education, and provide a risk stratification instrument for future research trials.

Fig 1. Flowchart of the populations used to build and externally validate mortality prediction models.

Fig 2. Optimal decision tree for categorizing patients admitted for COVID-19 based on the most informative predictors of 14-day mortality.

Fig 3. Discrimination and calibration of prediction models in internal validation.

a) 7-day mortality: Receiver operator characteristic (ROC) curve for discrimination. Area under the curve (AUC) = 0.877 (95%CI 0.831–0.923). b) 7-day mortality: Calibration plot of observed versus predicted risk of mortality (Hosmer-Lemeshow chi-squared = 9.10, p = 0.334; DF = 8). c) 14-day mortality: Receiver operator characteristic (ROC) curve for discrimination. Area under the curve (AUC) = 0.847 (95%CI 0.806–0.888). d) 14-day mortality: Calibration plot of observed versus predicted risk of mortality (Hosmer-Lemeshow chi-squared = 9.64, p = 0.291; DF = 8).

Fig 4. Discrimination and calibration of prediction models in external validation.

a) 7-day mortality: Receiver operator characteristic (ROC) curve for discrimination. Area under the curve (AUC) = 0.851 (95%CI 0.781–0.921). b) 7-day mortality: Calibration plot of observed versus predicted risk of mortality (Hosmer-Lemeshow chi-squared = 9.03, p = 0.340; DF = 8). c) 14-day mortality: Receiver operator characteristic (ROC) curve for discrimination. Area under the curve (AUC) = 0.825 (95%CI 0.764–0.887). d) 14-day mortality: Calibration plot of observed versus predicted risk of mortality (Hosmer-Lemeshow chi-squared = 7.63, p = 0.471; DF = 8).