Preintubation Sequential Organ Failure Assessment Score for Predicting COVID-19 Mortality: External Validation Using Electronic Health Record From 86 U.S. Healthcare Systems to Appraise Current Ventilator Triage Algorithms

Abstract

Objectives: Prior research has hypothesized the Sequential Organ Failure Assessment (SOFA) score to be a poor predictor of mortality in mechanically ventilated patients with COVID-19. Yet, several U.S. states have proposed SOFA-based algorithms for ventilator triage during crisis standards of care. Using a large cohort of mechanically ventilated patients with COVID-19, we externally validated the predictive capacity of the preintubation SOFA score for mortality prediction with and without other commonly used algorithm elements.

Design: Multicenter, retrospective cohort study using electronic health record data.

Setting: Eighty-six U.S. health systems.

Patients: Patients with COVID-19 hospitalized between January 1, 2020, and February 14, 2021, and subsequently initiated on mechanical ventilation.

Interventions: None.

Measurements and main results: Among 15,122 mechanically ventilated patients with COVID-19, SOFA score alone demonstrated poor discriminant accuracy for inhospital mortality in mechanically ventilated patients using the validation cohort (area under the receiver operating characteristic curve [AUC], 0.66; 95% CI, 0.65-0.67). Discriminant accuracy was even poorer using SOFA score categories (AUC, 0.54; 95% CI, 0.54-0.55). Age alone demonstrated greater discriminant accuracy for inhospital mortality than SOFA score (AUC, 0.71; 95% CI, 0.69-0.72). Discriminant accuracy for mortality improved upon addition of age to the continuous SOFA score (AUC, 0.74; 95% CI, 0.73-0.76) and categorized SOFA score (AUC, 0.72; 95% CI, 0.71-0.73) models, respectively. The addition of comorbidities did not substantially increase model discrimination. Of 36 U.S. states with crisis standards of care guidelines containing ventilator triage algorithms, 31 (86%) feature the SOFA score. Of these, 25 (81%) rely heavily on the SOFA score (12 exclusively propose SOFA; 13 place highest weight on SOFA or propose SOFA with one other variable).

Conclusions: In a U.S. cohort of over 15,000 ventilated patients with COVID-19, the SOFA score displayed poor predictive accuracy for short-term mortality. Our findings warrant reappraisal of the SOFA score's implementation and weightage in existing ventilator triage pathways in current U.S. crisis standards of care guidelines.

Figure 1.

Study flowchart depicting the exclusion of patients with do-not-resuscitate (DNR)/do-not-intubate (DNI) status and mechanical ventilation on within 24 hr of admission; 15,122 patients were included in the final analysis.

Figure 2.

Calibration belts for mortality prediction scores. A, Continuous Sequential Organ Failure Assessment (SOFA) score. B, SOFA score + age. The range of values for which the predicted mortality overestimates mortality (the observed mortality values are significantly under the bisector) or underestimates mortality (observed mortality lies above the bisector) based on the shaded 95% CI is reported at the bottom of each graph.

Figure 3.

Heat map illustrating the availability of crisis standards of care (CSC) protocols in the United States by state and degree of reliance on Sequential Organ Failure Assessment (SOFA) score to guide scarce resource allocation. States with COVID-specific guidelines are underlined.