Performance of intensive care unit severity scoring systems across different ethnicities in the USA: a retrospective observational study

Abstract

Background: Despite wide use of severity scoring systems for case-mix determination and benchmarking in the intensive care unit (ICU), the possibility of scoring bias across ethnicities has not been examined. Guidelines on the use of illness severity scores to inform triage decisions for allocation of scarce resources, such as mechanical ventilation, during the current COVID-19 pandemic warrant examination for possible bias in these models. We investigated the performance of the severity scoring systems Acute Physiology and Chronic Health Evaluation IVa (APACHE IVa), Oxford Acute Severity of Illness Score (OASIS), and Sequential Organ Failure Assessment (SOFA) across four ethnicities in two large ICU databases to identify possible ethnicity-based bias.

Methods: Data from the electronic ICU Collaborative Research Database (eICU-CRD) and the Medical Information Mart for Intensive Care III (MIMIC-III) database, built from patient episodes in the USA from 2014-15 and 2001-12, respectively, were analysed for score performance in Asian, Black, Hispanic, and White people after appropriate exclusions. Hospital mortality was the outcome of interest. Discrimination and calibration were determined for all three scoring systems in all four groups, using area under receiver operating characteristic (AUROC) curve for different ethnicities to assess discrimination, and standardised mortality ratio (SMR) or proxy measures to assess calibration.

Findings: We analysed 166 751 participants (122 919 eICU-CRD and 43 832 MIMIC-III). Although measurements of discrimination were significantly different among the groups (AUROC ranging from 0·86 to 0·89 [p=0·016] with APACHE IVa and from 0·75 to 0·77 [p=0·85] with OASIS), they did not display any discernible systematic patterns of bias. However, measurements of calibration indicated persistent, and in some cases statistically significant, patterns of difference between Hispanic people (SMR 0·73 with APACHE IVa and 0·64 with OASIS) and Black people (0·67 and 0·68) versus Asian people (0·77 and 0·95) and White people (0·76 and 0·81). Although calibrations were imperfect for all groups, the scores consistently showed a pattern of overpredicting mortality for Black people and Hispanic people. Similar results were seen using SOFA scores across the two databases.

Interpretation: The systematic differences in calibration across ethnicities suggest that illness severity scores reflect statistical bias in their predictions of mortality.

Funding: There was no specific funding for this study.

Figure 1:. Study flow

Excluded patients in both databases; the exclusions have been made in the sequence specified in the diagram. APACHE IVa=Acute Physiology and Chronic Health Evaluation IVa. eICU-CRD=electronic intensive care unit Collaborative Research Database. MIMIC-III=Medical Information Mart for Intensive Care III. SOFA=Sequential Organ Failure Assessment.

Figure 2:. ROC for predicted hospital mortality by ethnicity in the eICU-CRD

(A) ROC for the APACHE IVa-predicted hospital mortality in the eICU-CRD by ethnicity. The AUROC for all was 0·86, Hispanic 0·89, Black 0·87, White 0·86, and Asian 0·86. (B) ROC for the OASIS-predicted hospital mortality in the MIMIC-III database by ethnicity. The AUROC for all was 0·76, Hispanic 0·76, Black 0·75, White 0·76, and Asian 0·77. ROC=receiver operating curve. APACHE IVa=Acute Physiology and Chronic Health Evaluation scoring system IVa. eICU-CRD=electronic intensive care unit Collaborative Research Database. AUROC=area under receiver operating characteristic. OASIS=Oxford Acute Severity of Illness Score. MIMIC-III=Medical Information Mart for Intensive Care III.

Figure 3:. SMR for APACHE score in the eICU-CRD and OASIS score in MIMIC-III across ethnicities

(A) Forest plot for SMRs from the eICU-CRD for mortality predicted by APACHE IVa. (B) Forest plot for SMRs for different ethnicities from the MIMIC-III database for predicted mortality determined by OASIS. SMR=standardised mortality ratio. eICU-CRD=electronic intensive care unit Collaborative Research Database. APACHE IVa=Acute Physiology and Chronic Health Evaluation scoring system IVa. OASIS=Oxford Acute Severity of Illness Score.

Figure 4:. ROC curves for forest plots for predicted mortality by SOFA score in eICU-CRD and MIMIC-III

(A) ROC plots for all ethnicities in the eICU-CRD for SOFA score performance in hospital mortality prediction. The AUROC for all was 0·77, Hispanic 0·78, Black 0·79, White 0·77, and Asian 0·72. (B) ROC plots for all ethnicities in MIMIC-III for SOFA score performance in hospital mortality prediction. The AUROC for all was 0·73, Hispanic 0·74, Black 0·76, White 0·73, and Asian 0·73. (C) Forest plot for AUROCs in different ethnicities in the eICU-CRD for performance of SOFA score with 95% CIs. (D) Forest plot for AUROCs in different ethnicities in MIMIC-III for performance of SOFA score with 95% CIs. AUC=area under the curve. AUROC=area under receiver operating characteristic. eICU-CRD=electronic intensive care unit Collaborative Research Database. MIMIC-III=Medical Information Mart for Intensive Care-III. SOFA=Sequential Organ Failure and Assessment.