An artificial intelligence approach for predicting death or organ failure after hospitalization for COVID-19: development of a novel risk prediction tool and comparisons with ISARIC-4C, CURB-65, qSOFA, and MEWS scoring systems

Abstract

Background: We applied machine learning (ML) algorithms to generate a risk prediction tool [Collaboration for Risk Evaluation in COVID-19 (CORE-COVID-19)] for predicting the composite of 30-day endotracheal intubation, intravenous administration of vasopressors, or death after COVID-19 hospitalization and compared it with the existing risk scores.

Methods: This is a retrospective study of adults hospitalized with COVID-19 from March 2020 to February 2021. Patients, each with 92 variables, and one composite outcome underwent feature selection process to identify the most predictive variables. Selected variables were modeled to build four ML algorithms (artificial neural network, support vector machine, gradient boosting machine, and Logistic regression) and an ensemble model to generate a CORE-COVID-19 model to predict the composite outcome and compared with existing risk prediction scores. The net benefit for clinical use of each model was assessed by decision curve analysis.

Results: Of 1796 patients, 278 (15%) patients reached primary outcome. Six most predictive features were identified. Four ML algorithms achieved comparable discrimination (P > 0.827) with c-statistics ranged 0.849-0.856, calibration slopes 0.911-1.173, and Hosmer-Lemeshow P > 0.141 in validation dataset. These 6-variable fitted CORE-COVID-19 model revealed a c-statistic of 0.880, which was significantly (P < 0.04) higher than ISARIC-4C (0.751), CURB-65 (0.735), qSOFA (0.676), and MEWS (0.674) for outcome prediction. The net benefit of the CORE-COVID-19 model was greater than that of the existing risk scores.

Conclusion: The CORE-COVID-19 model accurately assigned 88% of patients who potentially progressed to 30-day composite events and revealed improved performance over existing risk scores, indicating its potential utility in clinical practice.

Keywords: COVID-19; Machine learning algorithms; Mortality; Organ failure; Prediction models.

Fig. 1

Schematics of data processing. A shows selected models with variable sets of the highest accuracies in ninety RFE procedures; the models involved in the RFE procedures were logistic regression, Naïve Bayes, and random forest; B illustrates number of times a variable was selected among the ninety RFE procedures; the count was the frequency for a feature to be chosen among the RFE procedures; C numbers of variables retained and tested in the RFE procedure in which the final chosen model was generated; the accuracy was the ratio of the number of correct predictions to the total number of predictions; D The variable importance level of the chosen model concerning the first nineteen features; the importance was the scaled score of the variable importance for the linear model. Abbreviations. ACE, angiotensin converting enzyme; ICU, intensive care unit; SSRI, selective serotonin receptor inhibitors

Fig. 2

Receiver operating characteristic curves (ROC) for predicting the composite of death or organ failure at 30 days after hospitalization for COVID-19. (A) development and (B) internal validation datasets stratified according to individual machine learning models; Fig. 2C shows ROC for predicting outcome stratified by the new CORE-COVID-19 and 4 existing risk prediction tools. CORE-COVI-19 model consistently outperformed each existing risk prediction tools. Fig. 2D and E showed decision curve analysis stratified according to machine learning models in development (D) and validation (E) data sets. Fig. 2F illustrate decision curve analysis stratified by CORE-COVID-19 and other existing risk prediction tools for outcome prediction with net benefit of CORE-COVID-19 exceeding that of other models at wide range of thresholds. The "intervention for all" indicated net benefit from 0 to 0.15 below 20% of threshold probability. The ML models achieved the best net benefit at around .07–.08 when the threshold probability approached the minimum in the training dataset. The models still showed net benefit when the threshold probability rose to approximately 75%; the GBM even showed net benefit at above 80% of threshold probability. On the validation data set, the best net benefit ranged between .05–.07, and the models offered net benefit at around 70% of threshold probability at most. The maximum net benefit for CORE-COVID-19 model was best at 0.1 threshold and continued to show net benefit at above 55% of threshold probability which was higher than existing prediction tools. ISARIC-4C had its best net benefit, which was comparable to ML models in training, but the maximum threshold probability showing net benefit was only around 35%. The qSOFA presents net benefit at above 50% of threshold probability but its best net benefit was only approximately 0.03. Abbreviations: AUC, area under receiver operating characteristic curve; CORE-COVID-19, Collaboration for Risk Evaluation in COVID-19; CURB-65, confusion, urea, respiratory rate, blood pressure, and age ≥ 65 years; ISARIC-4C, International Severe Acute Respiratory and emerging Infections Consortium Coronavirus Clinical Characterization Consortium; qSOFA, quick sequential organ failure assessment; MEWS, modified early warning score

Fig. 3

Calibration plots associated with each machine learning model in development (upper panel A–E) and validation (lower panel, A–E) datasets, all showed good calibration

Fig. 4

Kaplan-Meir estimates for cumulative incidence of the composite of death or organ failure by the tertiles of COVID-19 organ failure CORE-COVID-19 scores: low, intermediate, and high-risk. In cumulative cohort of 1794 hospitalized COVID-19 patients, 42.5% composite events occurred in highest compared with 7.9% in the intermediate and 1.4% in the lowest tertile. Hazard ratios and 95% confidence intervals were adjusted to demographics. Abbreviations. aHR, adjusted hazard ratio; CI confidence interval