Evaluation of ICU Risk Models Adapted for Use as Continuous Markers of Severity of Illness Throughout the ICU Stay

Abstract

Objectives: Evaluate the accuracy of different ICU risk models repurposed as continuous markers of severity of illness.

Design: Nonintervention cohort study.

Setting: eICU Research Institute ICUs using tele-ICU software calculating continuous ICU Discharge Readiness Scores between January 2013 and March 2016.

Patients: Five hundred sixty-one thousand four hundred seventy-eight adult ICU patients with an ICU length of stay between 4 hours and 30 days.

Interventions: Not available.

Measurements and main results: Hourly Acute Physiology and Chronic Health Evaluation IV, Sequential Organ Failure Assessment, and Discharge Readiness Scores were calculated beginning hour 4 of the ICU stay. Primary outcome was the area under the receiver operating characteristic curve for the mean score with ICU mortality. Secondary outcomes included area under the receiver operating characteristic curves for ICU mortality with admission, median, maximum and last scores, and for death within 24 hours. The trajectories of each score were visualized by plotting the hourly averages against time in the ICU, stratified by mortality and length of stay. The area under the receiver operating characteristic curves for mean Acute Physiology and Chronic Health Evaluation, Sequential Organ Failure Assessment, and Discharge Readiness Scores were 0.90 (0.89-0.90), 0.86 (0.86-0.86), and 0.94 (0.94-0.94), respectively. The area under the receiver operating characteristic curves for hourly Acute Physiology and Chronic Health Evaluation, Sequential Organ Failure Assessment, and Discharge Readiness Scores predicting 24-hour mortality were 0.81 (0.81-0.81), 0.76 (0.76-0.76), and 0.86 (0.86-0.86). Discharge Readiness Scores had a higher area under the receiver operating characteristic curve than both Acute Physiology and Chronic Health Evaluation and Sequential Organ Failure Assessment for each metric. Acute Physiology and Chronic Health Evaluation and Sequential Organ Failure Assessment scores increased throughout the first 24 hours in both survivors and nonsurvivors; Discharge Readiness Scores continuously decreased in survivors and temporarily decreased before increasing by hour 36 in nonsurvivors with longer length of stays.

Conclusions: Acute Physiology and Chronic Health Evaluation, Sequential Organ Failure Assessment, and Discharge Readiness Scores all have relatively high discrimination for ICU mortality when used continuously; Discharge Readiness Scores tended to have slightly higher area under the receiver operating characteristic curves for each endpoint. These findings validate the use of these models on a population level for continuous risk adjustment in the ICU, although Acute Physiology and Chronic Health Evaluation and Sequential Organ Failure Assessment appear slower to respond to improvements in patient status than Discharge Readiness Scores, and Discharge Readiness Scores may reflect physiologic improvement from interventions, potentially underestimating risk.