Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU

Abstract

Background: Patients in general medical-surgical wards who experience unplanned transfer to the intensive care unit (ICU) show evidence of physiologic derangement 6-24h prior to their deterioration. With increasing availability of electronic medical records (EMRs), automated early warning scores (EWSs) are becoming feasible.

Objective: To describe the development and performance of an automated EWS based on EMR data.

Materials and methods: We used a discrete-time logistic regression model to obtain an hourly risk score to predict unplanned transfer to the ICU within the next 12h. The model was based on hospitalization episodes from all adult patients (18years) admitted to 21 Kaiser Permanente Northern California (KPNC) hospitals from 1/1/2010 to 12/31/2013. Eligible patients met these entry criteria: initial hospitalization occurred at a KPNC hospital; the hospitalization was not for childbirth; and the EMR had been operational at the hospital for at least 3months. We evaluated the performance of this risk score, called Advanced Alert Monitor (AAM) and compared it against two other EWSs (eCART and NEWS) in terms of their sensitivity, specificity, negative predictive value, positive predictive value, and area under the receiver operator characteristic curve (c statistic).

Results: A total of 649,418 hospitalization episodes involving 374,838 patients met inclusion criteria, with 19,153 of the episodes experiencing at least one outcome. The analysis data set had 48,723,248 hourly observations. Predictors included physiologic data (laboratory tests and vital signs); neurological status; severity of illness and longitudinal comorbidity indices; care directives; and health services indicators (e.g. elapsed length of stay). AAM showed better performance compared to NEWS and eCART in all the metrics and prediction intervals. The AAM AUC was 0.82 compared to 0.79 and 0.76 for eCART and NEWS, respectively. Using a threshold that generated 1 alert per day in a unit with a patient census of 35, the sensitivity of AAM was 49% (95% CI: 47.6-50.3%) compared to the sensitivities of eCART and NEWS scores of 44% (42.3-45.1) and 40% (38.2-40.9), respectively. For all three scores, about half of alerts occurred within 12h of the event, and almost two thirds within 24h of the event.

Conclusion: The AAM score is an example of a score that takes advantage of multiple data streams now available in modern EMRs. It highlights the ability to harness complex algorithms to maximize signal extraction. The main challenge in the future is to develop detection approaches for patients in whom data are sparser because their baseline risk is lower.

Keywords: Critical care; Deterioration; Early warning score; Electronic health records; Patient safety; Physiologic monitoring; Risk score.

Fig. 1

Mean value of the AAM score in the 24 h prior to event. The figure compares score trajectories among episodes where deterioration (outcome defined in text) did and did not occur. Episodes where the outcome occurred are shown at top (upper points and grey fitted line). For episodes without an outcome (lower points and dashed fitted line), we selected a random 24 h period. Episodes without the outcome have generally lower scores than episodes with the outcome. The figure shows that the AAM score starts increasing about 8 h prior to the event with the average score being close to 12 at the time of the event.

Fig. 2

Percent of alerts triggered by hours between alert and event for AAM, eCART and NEWS. This figure shows the distribution of alerts triggered by hours between the alert and the event for AAM, eCART and NEWS based on the Validation data set for episodes experiencing an event. An alert was triggered if the score was greater than the training cutoff. Each respective score’s training cutoff was determined so that there would be no more than 1 alert per day in a hospital with an average daily census of ~35 patients. The cutoff used for eCART was 50, for NEWS was 8 and for AAM was 7.5.

Fig. 3

eCART, NEWS and AAM receiver operator curves. This figure shows the Receiver Operating Characteristic Curve (ROC) for all three scores using the episode-based Validation data. The ROC is a standard technique for summarizing a classification model’s discrimination performance. The graph shows each score’s sensitivity (true-positive rate) vs. 1 minus sensitivity (false-positive rate) across all cutoffs. The c-statistic or the area under the ROC curve (AUC) measures the probability that given two patients (one who experienced the event and one who did not), the model will assign a higher score to the former (see citation 32). Given that there is no consensus over the use of an hourly data structure (with repeat alerts) vs. an episode-based structure for evaluating model performance, we calculated the c-statistic using both the hourly and the episode-based structured data. Both the curves and the c-statistics show that AAM has better discrimination ability than eCART, which performs better than NEWS, across all cutoffs.

Fig. 4

eCART, NEWS and AAM precision-recall curves. Precision-Recall (PR) curves of all three scores described in this paper. The PR curves show the scores’ positive predictive value (PPV or Precision) against the sensitivity (recall). PR curves are commonly used to present results of binary classifiers of outcomes when the outcome variable is extremely rare. The PPV or Precision is the inverse of the work-up to detection ratio (W:D). The figure shows that the AAM has higher sensitivity than eCART and NEWS for any given PPV or W:D ratio.

Fig. 5

eCART, NEWS and AAM sensitivity vs. alerts per day. The figure shows the sensitivity of each score against the number of alerts per day in a hospital with an average daily census of 70 patients as the cutoffs move across the scores’ range. The AAM score has higher sensitivity than eCART and NEWS across all alerts per day.

Fig. 6

eCART, NEWS and AAM specificity vs. alerts per day. The figure shows the specificity of each score against the number of alerts per day in a hospital with an average daily census of 70 patients as the cutoffs move across the scores’ range. All three scores have similar specificity.