A Comparative Analysis of the Respiratory Subscore of the Sequential Organ Failure Assessment Scoring System

Abstract

Rationale: The Sequential Organ Failure Assessment (SOFA) tool is a commonly used measure of illness severity. Calculation of the respiratory subscore of SOFA is frequently limited by missing arterial oxygen pressure (Pa_O2) data. Although missing Pa_O2 data are commonly replaced with normal values, the performance of different methods of substituting Pa_O2 for SOFA calculation is unclear. Objectives: The study objective was to compare the performance of different substitution strategies for missing Pa_O2 data for SOFA score calculation. Methods: This retrospective cohort study was performed using the Weill Cornell Critical Care Database for Advanced Research from a tertiary care hospital in the United States. All adult patients admitted to an intensive care unit (ICU) from 2011 to 2019 with an available respiratory SOFA score were included. We analyzed the availability of the Pa_O2/fraction of inspired oxygen (Fi_O2) ratio on the first day of ICU admission. In those without a Pa_O2/Fi_O2 ratio available, the ratio of oxygen saturation as measured by pulse oximetry to Fi_O2 was used to calculate a respiratory SOFA subscore according to four methods (linear substitution [Rice], nonlinear substitution [Severinghaus], modified respiratory SOFA, and multiple imputation by chained equations [MICE]) as well as the missing-as-normal technique. We then compared how well the different total SOFA scores discriminated in-hospital mortality. We performed several subgroup and sensitivity analyses. Results: We identified 35,260 unique visits, of which 9,172 included predominant respiratory failure. Pa_O2 data were available for 14,939 (47%). The area under the receiver operating characteristic curve for each substitution technique for discriminating in-hospital mortality was higher than that for the missing-as-normal technique (0.78 [0.77-0.79]) in all analyses (modified, 0.80 [0.79-0.81]; Rice, 0.80 [0.79-0.81]; Severinghaus, 0.80 [0.79-0.81]; and MICE, 0.80 [0.79-0.81]) (P < 0.01). Each substitution method had a higher accuracy for discriminating in-hospital mortality (MICE, 0.67; Rice, 0.67; modified, 0.66; and Severinghaus, 0.66) than the missing-as-normal technique. Model calibration for in-hospital mortality was less precise for the missing-as-normal technique than for the other substitution techniques at the lower range of SOFA and among the subgroups. Conclusions: Using physiologic and statistical substitution methods improved the total SOFA score's ability to discriminate mortality compared with the missing-as-normal technique. Treating missing data as normal may result in underreporting the severity of illness compared with using substitution. The simplicity of a direct oxygen saturation as measured by pulse oximetry/Fi_O2 ratio-modified SOFA technique makes it an attractive choice for electronic health record-based research. This knowledge can inform comparisons of severity of illness across studies that used different techniques.

Keywords: Sequential Organ Failure Assessment score; imputation; organ dysfunction; respiratory failure; survival.

Figure 1.

Alluvial plot visualizing the relationship between the respiratory SOFA subscore by each imputation technique. (A) The total Weill Cornell Critical Care Database for Advanced Research population is displayed. (B and C) Patients without an Pa_O2 value and with predominant respiratory failure are shown. Patients are grouped and color coded by the assigned value from the missing-as-normal substitution technique in the leftmost column and arranged by subscore. Bands connect each imputation technique from left to right (missing-as-normal technique to modified to Rice to Severinghaus to MICE). A thicker band indicates that a greater number of patients have a particular classification by that substitution technique. MICE = multiple imputation by chained equations; Pa_O2 = arterial oxygen pressure; SOFA = Sequential Organ Failure Assessment.

Figure 2.

Comparative receiver operating characteristic curves for each substitution technique predicting in-hospital mortality for (A) the total population, (C) patients without an arterial oxygen pressure (Pa_O2) value, and (E) patients with predominant respiratory failure, respectively. (B) The area under the receiver operating characteristic curve (AUC) and 95% confidence interval (CI) for the total population: missing-as-normal technique, 0.78 (0.77–0.79); modified, 0.80 (0.79–0.81); Rice, 0.80 (0.79–0.81); Severinghaus, 0.80 (0.79–0.81); and multiple imputation by chained equations (MICE), 0.80 (0.79–0.81). (D) The AUC for patients without a Pa_O2 value: missing-as-normal technique, 0.83 (0.81–0.85); modified, 0.86 (0.85–0.88); Rice, 0.860 (0.8–0.87); Severinghaus, 0.85 (0.84–0.87); and MICE, 0.86 (0.85–0.87). (F) The AUC for patients with predominant respiratory failure: missing-as-normal technique, 0.73 (0.71–0.76); modified, 0.77 (0.75–0.79); Rice, 0.76 (0.74–0.79); Severinghaus, 0.76 (0.74–0.79); and MICE, 0.76 (0.74–0.79). All CIs were computed by bootstrapping, and for MICE, the standard errors were pooled using Rubin rules (46). SOFA = Sequential Organ Failure Assessment.

Figure 3.

Calibration plots for the (A) missing-as-normal technique, (B) modified, (C) Rice, (D) Severinghaus, and (E) MICE substitution methods. Black circles indicate the in-hospital mortality rate and corresponding 95% confidence interval for each decile of predicted probability. The solid line shows perfect calibration, and the dashed line shows the least squares fit through the points. The red line shows a moving average for the observed mortality rates across the predicted probabilities. All plots are truncated at a predicted probability of 0.5 to allow differences among substitution methods to be observed more clearly. MICE = multiple imputation by chained equations.

Figure 4.

(A–C) The proportions of in-hospital mortality among total Sequential Organ Failure Assessment (SOFA) scores after using each respiratory SOFA substitution technique as well as the missing-as-normal technique among the total cohort (A), patients without an Pa_O2 value (B), and patients with predominant respiratory failure (C), respectively. Total SOFA scores with low sample sizes in the population were collapsed into a high SOFA category. MICE = multiple imputation by chained equations; Pa_O2= arterial oxygen pressure.