Detecting Deteriorating Patients in the Hospital: Development and Validation of a Novel Scoring System

Abstract

Rationale: Late recognition of patient deterioration in hospital is associated with worse outcomes, including higher mortality. Despite the widespread introduction of early warning score (EWS) systems and electronic health records, deterioration still goes unrecognized. Objectives: To develop and externally validate a Hospital- wide Alerting via Electronic Noticeboard (HAVEN) system to identify hospitalized patients at risk of reversible deterioration. Methods: This was a retrospective cohort study of patients 16 years of age or above admitted to four UK hospitals. The primary outcome was cardiac arrest or unplanned admission to the ICU. We used patient data (vital signs, laboratory tests, comorbidities, and frailty) from one hospital to train a machine-learning model (gradient boosting trees). We internally and externally validated the model and compared its performance with existing scoring systems (including the National EWS, laboratory-based acute physiology score, and electronic cardiac arrest risk triage score). Measurements and Main Results: We developed the HAVEN model using 230,415 patient admissions to a single hospital. We validated HAVEN on 266,295 admissions to four hospitals. HAVEN showed substantially higher discrimination (c-statistic, 0.901 [95% confidence interval, 0.898-0.903]) for the primary outcome within 24 hours of each measurement than other published scoring systems (which range from 0.700 [0.696-0.704] to 0.863 [0.860-0.865]). With a precision of 10%, HAVEN was able to identify 42% of cardiac arrests or unplanned ICU admissions with a lead time of up to 48 hours in advance, compared with 22% by the next best system. Conclusions: The HAVEN machine-learning algorithm for early identification of in-hospital deterioration significantly outperforms other published scores such as the National EWS.

Keywords: early warning scores; machine learning; physiological monitoring; retrospective observational study.

Figure 1.

Calibration curve for Hospital-wide Alerting via Electronic Noticeboard predictions in the test set (top), alongside the distribution of Hospital-wide Alerting via Electronic Noticeboard predictions (bottom). The calibration curve (black) shows the locally estimated (scatterplot smoothing) smoothed observed probability versus estimated probability of adverse events (with 95% confidence bands). The diagonal line (gray) shows ideal calibration.

Figure 2.

Patient admission level sensitivity: average proportion of (candidate) adverse events to be identified within each window (left); and the average proportion of adverse events identified ahead of time for Hospital-wide Alerting via Electronic Noticeboard at different precision levels (5%, 10%, and 20%) (right). The error bars denote 1 SD.