Validation of computerized automatic calculation of the sequential organ failure assessment score

Abstract

Purpose. To validate the use of a computer program for the automatic calculation of the sequential organ failure assessment (SOFA) score, as compared to the gold standard of manual chart review. Materials and Methods. Adult admissions (age > 18 years) to the medical ICU with a length of stay greater than 24 hours were studied in the setting of an academic tertiary referral center. A retrospective cross-sectional analysis was performed using a derivation cohort to compare automatic calculation of the SOFA score to the gold standard of manual chart review. After critical appraisal of sources of disagreement, another analysis was performed using an independent validation cohort. Then, a prospective observational analysis was performed using an implementation of this computer program in AWARE Dashboard, which is an existing real-time patient EMR system for use in the ICU. Results. Good agreement between the manual and automatic SOFA calculations was observed for both the derivation (N=94) and validation (N=268) cohorts: 0.02 ± 2.33 and 0.29 ± 1.75 points, respectively. These results were validated in AWARE (N=60). Conclusion. This EMR-based automatic tool accurately calculates SOFA scores and can facilitate ICU decisions without the need for manual data collection. This tool can also be employed in a real-time electronic environment.

Figure 1

Derivation cohort: Bland-Altman plot of the manual versus automatic scores resulted in a mean difference of 0.02 ± 2.33 (SD, N = 94). Note: plotted values frequently represent more than one patient sample.

Figure 2

Validation cohort: Bland-Altman plot of the manual versus automatic scores resulted in a mean difference of 0.12 ± 1.64 (SD, N = 268). Note: plotted values frequently represent more than one patient sample.

Figure 3

Derivation cohort: sum of the absolute value of the difference between the automatic and manual SOFA component scores. For the outlier box plot, the first box (red bracket) represents the first quartile, followed by the second and third quartiles. Fourth quartile outliers are represented by black dots. The triangle represents the mean difference (1.64).

Figure 4

Validation cohort: sum of the absolute value of the difference between the automatic and manual SOFA component scores. For the outlier box plot, the first box represents the first and second quartiles, followed by the third quartile. Fourth quartile outliers are represented by black dots. The triangle represents the mean difference (0.85).