Multicenter development and validation of a risk stratification tool for ward patients

Abstract

Rationale: Most ward risk scores were created using subjective opinion in individual hospitals and only use vital signs.

Objectives: To develop and validate a risk score using commonly collected electronic health record data.

Methods: All patients hospitalized on the wards in five hospitals were included in this observational cohort study. Discrete-time survival analysis was used to predict the combined outcome of cardiac arrest (CA), intensive care unit (ICU) transfer, or death on the wards. Laboratory results, vital signs, and demographics were used as predictor variables. The model was developed in the first 60% of the data at each hospital and then validated in the remaining 40%. The final model was compared with the Modified Early Warning Score (MEWS) using the area under the receiver operating characteristic curve and the net reclassification index (NRI).

Measurements and main results: A total of 269,999 patient admissions were included, with 424 CAs, 13,188 ICU transfers, and 2,840 deaths occurring during the study period. The derived model was more accurate than the MEWS in the validation dataset for all outcomes (area under the receiver operating characteristic curve, 0.83 vs. 0.71 for CA; 0.75 vs. 0.68 for ICU transfer; 0.93 vs. 0.88 for death; and 0.77 vs. 0.70 for the combined outcome; P value < 0.01 for all comparisons). This accuracy improvement was seen across all hospitals. The NRI for the electronic Cardiac Arrest Risk Triage compared with the MEWS was 0.28 (0.18-0.38), with a positive NRI of 0.19 (0.09-0.29) and a negative NRI of 0.09 (0.09-0.09).

Conclusions: We developed an accurate ward risk stratification tool using commonly collected electronic health record variables in a large multicenter dataset. Further study is needed to determine whether implementation in real-time would improve patient outcomes.

Keywords: decision support techniques; early diagnosis; heart arrest; hospital rapid response team; statistical models.

Figure 1.

Areas under the receiver operating characteristic curves (AUCs) for the Modified Early Warning Score (MEWS) and electronic Cardiac Arrest Risk Triage (eCART) score for whether an event occurred within 24 hours of an observation. ICU = intensive care unit.

Figure 2.

Shown is the percentage of patients who reached each electronic Cardiac Arrest Risk Triage (eCART) score during their ward admission in the validation cohort, with each outcome and the entire study population shown as a separate line. ICU = intensive care unit.

Figure 3.

Change over time of the mean electronic Cardiac Arrest Risk Triage (eCART) score in the 24 hours before cardiac arrest, ICU transfer, and death and a random 24-hour period for those patients who experienced neither event. ICU = intensive care unit.