Development and Validation of the Quick COVID-19 Severity Index: A Prognostic Tool for Early Clinical Decompensation

Abstract

Study objective: The goal of this study is to create a predictive, interpretable model of early hospital respiratory failure among emergency department (ED) patients admitted with coronavirus disease 2019 (COVID-19).

Methods: This was an observational, retrospective, cohort study from a 9-ED health system of admitted adult patients with severe acute respiratory syndrome coronavirus 2 (COVID-19) and an oxygen requirement less than or equal to 6 L/min. We sought to predict respiratory failure within 24 hours of admission as defined by oxygen requirement of greater than 10 L/min by low-flow device, high-flow device, noninvasive or invasive ventilation, or death. Predictive models were compared with the Elixhauser Comorbidity Index, quick Sequential [Sepsis-related] Organ Failure Assessment, and the CURB-65 pneumonia severity score.

Results: During the study period, from March 1 to April 27, 2020, 1,792 patients were admitted with COVID-19, 620 (35%) of whom had respiratory failure in the ED. Of the remaining 1,172 admitted patients, 144 (12.3%) met the composite endpoint within the first 24 hours of hospitalization. On the independent test cohort, both a novel bedside scoring system, the quick COVID-19 Severity Index (area under receiver operating characteristic curve mean 0.81 [95% confidence interval {CI} 0.73 to 0.89]), and a machine-learning model, the COVID-19 Severity Index (mean 0.76 [95% CI 0.65 to 0.86]), outperformed the Elixhauser mortality index (mean 0.61 [95% CI 0.51 to 0.70]), CURB-65 (0.50 [95% CI 0.40 to 0.60]), and quick Sequential [Sepsis-related] Organ Failure Assessment (0.59 [95% CI 0.50 to 0.68]). A low quick COVID-19 Severity Index score was associated with a less than 5% risk of respiratory decompensation in the validation cohort.

Conclusion: A significant proportion of admitted COVID-19 patients progress to respiratory failure within 24 hours of admission. These events are accurately predicted with bedside respiratory examination findings within a simple scoring system.

Figure 1

Model development strategy. Exclusions were for critical illness within 4 hours of ED presentation.

Figure 2

SHAP variable importance and bee swarm plots. A, Mean absolute SHAP values suggest a rank order for variable importance in the COVID-19 Severity Index. B, In the bee swarm plot, each point corresponds to an individual person in the study. The points’ position on the x axis shows the effect that feature has on the model’s prediction for a given patient. Color corresponds to relative variable value.

Figure 3

SHAP value plots for age (A), alanine aminotransferase (B), aspartate aminotransferase (C), and ferritin (D). Scatter plots show the effects of variable values (x axis) on the model predictions as captured by SHAP values (y axis).

Figure 4

Calibration of quick COVID-19 Severity Index and COVID-19 Severity Index on the independent validation data set. A, Each patient in the validation cohort was assigned a score by quick COVID-19 Severity Index, and the percentage who had a critical respiratory illness outcome were plotted with a line plot. Patients were then grouped into risk bins by quick COVID-19 Severity Index intervals (0 to 3, 4 to 6, 7 to 9, and 10 to 12); the percentage of patients in each group with the outcome is indicated in the bar plot. B, Each patient in the validation cohort was assigned a COVID-19 Severity Index score, a percentage risk from 0% to 100% using gradient boosting and isotonic regression. The percentage of patients with COVID-19 Severity Index scores of 0% to 33%, 33% to 66%, and 66% to 100% who experienced critical respiratory illness at 24 hours is shown.