Artificial intelligence-based model for COVID-19 prognosis incorporating chest radiographs and clinical data; a retrospective model development and validation study

Abstract

Objectives: The purpose of this study was to develop an artificial intelligence-based model to prognosticate COVID-19 patients at admission by combining clinical data and chest radiographs.

Methods: This retrospective study used the Stony Brook University COVID-19 dataset of 1384 inpatients. After exclusions, 1356 patients were randomly divided into training (1083) and test datasets (273). We implemented three artificial intelligence models, which classified mortality, ICU admission, or ventilation risk. Each model had three submodels with different inputs: clinical data, chest radiographs, and both. We showed the importance of the variables using SHapley Additive exPlanations (SHAP) values.

Results: The mortality prediction model was best overall with area under the curve, sensitivity, specificity, and accuracy of 0.79 (0.72-0.86), 0.74 (0.68-0.79), 0.77 (0.61-0.88), and 0.74 (0.69-0.79) for the clinical data-based model; 0.77 (0.69-0.85), 0.67 (0.61-0.73), 0.81 (0.67-0.92), 0.70 (0.64-0.75) for the image-based model, and 0.86 (0.81-0.91), 0.76 (0.70-0.81), 0.77 (0.61-0.88), 0.76 (0.70-0.81) for the mixed model. The mixed model had the best performance (p value < 0.05). The radiographs ranked fourth for prognostication overall, and first of the inpatient tests assessed.

Conclusions: These results suggest that prognosis models become more accurate if AI-derived chest radiograph features and clinical data are used together.

Advances in knowledge: This AI model evaluates chest radiographs together with clinical data in order to classify patients as having high or low mortality risk. This work shows that chest radiographs taken at admission have significant COVID-19 prognostic information compared to clinical data other than age and sex.

Figure 1.

Eligibility flowchart.

Figure 2.

Overview of the classification and prognostic models. The chest radiographs were input as PNG images, and the clinical data were in a CSV file. Three sets of models were created; one each with the output arranged to show if the COVID-19 patient was expected to die, require mechanical ventilation, or require ICU admission. FC = Fully connected layers

Figure 3.

Receiver operating characteristic curves. Each panel represents different outcome targets for the models. (a) Risk of death. The clinical data-based model had an AUC of 0.79 (0.72–0.86), the image-based model had an AUC of 0.77 (0.69–0.85), and the mixed model had an AUC of 0.86 (0.81–0.91). (b) Risk of mechanical ventilation. The clinical data-based model had an AUC of 0.70 (0.62–0.77), the image-based model had an AUC of 0.68 (0.60–0.77), and the mixed model had an AUC of 0.76 (0.69–0.83). (c) Risk of ICU admission. The clinical data-based model had an AUC of 0.68 (0.61–0.75), the image-based model had an AUC of 0.70 (0.62–0.77), and the mixed model had an AUC of 0.78 (0.72–0.84). AUC = area under the curve

Figure 4.

Kaplan–Meier survival plots for each model. The high-risk and low-risk patients from each mortality model were divided based on the median model output value. This plot shows the ground truth survival of these patients and the shaded area represents the accuracy of the prediction.

Figure 5.

SHAP values for each variable. Beeswarm plots of the SHAP value for each patient for the top ten variables. The plots relate to risk of death (a), risk of mechanical ventilation (b), and risk of ICU admission (c). Each dot represents one patient. The location of the dot represents if changing the value for that patient would have a positive (less likely to predict outcome) or negative (more likely to predict outcome) effect on the model, and to what extent. The color represents the value range of the variable from lowest (blue) to highest (red). Some variables only have binomial representations; for sex, red represents male. For age, red represents an age above the cutoff. When there are many patients with very similar SHAP values, the swarm expands vertically.

Figure 6.

Representative saliency maps. These are chest radiographs and the saliency map overlay of two patients (labeled a and b) from the Stony Brook Hospital dataset. The mortality prediction model was used. In these images, the model focus was on an area of infiltration.