Clinical features of COVID-19 mortality: development and validation of a clinical prediction model

Abstract

Background: The COVID-19 pandemic has affected millions of individuals and caused hundreds of thousands of deaths worldwide. Predicting mortality among patients with COVID-19 who present with a spectrum of complications is very difficult, hindering the prognostication and management of the disease. We aimed to develop an accurate prediction model of COVID-19 mortality using unbiased computational methods, and identify the clinical features most predictive of this outcome.

Methods: In this prediction model development and validation study, we applied machine learning techniques to clinical data from a large cohort of patients with COVID-19 treated at the Mount Sinai Health System in New York City, NY, USA, to predict mortality. We analysed patient-level data captured in the Mount Sinai Data Warehouse database for individuals with a confirmed diagnosis of COVID-19 who had a health system encounter between March 9 and April 6, 2020. For initial analyses, we used patient data from March 9 to April 5, and randomly assigned (80:20) the patients to the development dataset or test dataset 1 (retrospective). Patient data for those with encounters on April 6, 2020, were used in test dataset 2 (prospective). We designed prediction models based on clinical features and patient characteristics during health system encounters to predict mortality using the development dataset. We assessed the resultant models in terms of the area under the receiver operating characteristic curve (AUC) score in the test datasets.

Findings: Using the development dataset (n=3841) and a systematic machine learning framework, we developed a COVID-19 mortality prediction model that showed high accuracy (AUC=0·91) when applied to test datasets of retrospective (n=961) and prospective (n=249) patients. This model was based on three clinical features: patient's age, minimum oxygen saturation over the course of their medical encounter, and type of patient encounter (inpatient vs outpatient and telehealth visits).

Interpretation: An accurate and parsimonious COVID-19 mortality prediction model based on three features might have utility in clinical settings to guide the management and prognostication of patients affected by this disease. External validation of this prediction model in other populations is needed.

Funding: National Institutes of Health.

Figure 1

Workflow for data management and COVID-19 mortality prediction model development Data were obtained from the Mount Sinai Data Warehouse. After pre-processing, data for patients with COVID-19 (n=4802) were randomly divided in an 80:20 ratio into a prediction model development dataset (n=3841) and an independent retrospective validation dataset (test dataset 1; n=961). For prediction model training and selection, the development dataset was further randomly split into a 75% training dataset (n=2880) and a 25% holdout dataset (n=961). Four classification algorithms were assessed. The final predictive model was validated on test dataset 1 and another independent prospective validation dataset (test dataset 2; n=249). LR=logistic regression. RF=random forest. SVM=support vector machine. XGBoost=eXtreme Gradient Boosting.

Figure 2

Results from missing value imputation (A) and feature selection (B) during prediction model training and selection (A) Datapoints show the average AUC score for each candidate algorithm and missing value level, with error bars shown by whiskers. (B) Datapoints show the average AUC score for each subset of number of features with error bars shown by whiskers. The details of the computational methods underlying these analyses are provided in the appendix (pp 2–3). AUC=area under the receiver operating characteristic curve. XGBoost=eXtreme Gradient Boosting.

Figure 3

Performance of the mortality prediction models on two validation datasets Evaluation results for test datasets 1 (A) and 2 (B) are shown here in terms of the ROC curves obtained, as well as their AUC scores, with 95% CIs in parentheses. Calibration curves of the 3F and 17F models on test datasets 1 (C) and 2 (D), with the slopes and intercepts of all the curves, along with their 95% CIs in parentheses. AUC=area under the ROC curve. ROC=receiver operating characteristic.

Figure 4

Top predictive features selected for the four classification algorithms (A) Top three predictive features identified using the recursive feature elimination method for the four classification algorithms across the 100 runs used to select the most discriminative features and train the corresponding candidate prediction models; the values in parentheses indicate the number of times the feature was selected as top ranked in the development dataset. Minimum oxygen saturation (B) and age (C) features, which were selected as top predictive features for all the four algorithms, are presented as violin plots showing the distributions of the values in the development dataset. In panels B and C, the black boxplots in the middle show the distribution of the values on the y axis, with the white dot indicating the median value; the width of the grey shape at a given value on the y axis indicates the probability of occurrence of that value in the population shown. The plots in panel B show that the median value (79%) of minimum oxygen saturation for the deceased group was significantly lower (Student's t test p<0·0001) than the median value (92%) for the alive group. Similarly, the plots in panel C show that the median age (75 years) in the deceased group is higher (Student's t test p<0·0001) than that in the alive group (56 years). COPD=chronic obstructive pulmonary disease.