Postimplementation Evaluation of a Machine Learning-Based Deterioration Risk Alert to Enhance Sepsis Outcome Improvements

Abstract

Machine learning-based early warning systems (EWSs) can detect clinical deterioration more accurately than point-score tools. In patients with sepsis, however, the timing and scope of sepsis interventions relative to an advanced EWS alert are not well understood. The objectives of this study were to evaluate the timing and frequency of fluid bolus therapy, new antibiotics, and Do Not Resuscitate (DNR) status relative to the time of an advanced EWS alert. We conducted 2 rounds of chart reviews of patients with an EWS alert admitted to community hospitals of a large integrated health system in Northern California (round 1: n = 21; round 2: n = 47). We abstracted patient characteristics and process measures of sepsis intervention and performed summary statistics. Sepsis decedents were older and sicker at admission and alert time. Most EWS alerts occurred near admission, and most sepsis interventions occurred before the first alert. Of 14 decedents, 12 (86%) had a DNR order before death. Fluid bolus therapy and new intravenous antibiotics frequently occurred before the alert, suggesting a potential overlap between sepsis care in the emergency department and the first alert following admission. Two tactics to minimize alerts that may not motivate new sepsis interventions are (1) locking out the alert during the immediate time after hospital admission; and (2) triaging and reviewing patients with alerts outside of the unit before activating a bedside response. Some decedents may have been on a palliative/end-of-life trajectory, because DNR orders were very common among decedents. Nurse leaders sponsoring or leading machine learning projects should consider tactics to reduce false-positive and clinically meaningless alerts dispatched to clinical staff.

Figure 1.

Hospitalization time markers for patients with sepsis flagged as at risk by an advanced early warning system. Note: Not to scale; for illustration purposes only; the figure shows the first wave of sepsis treatment, with overlap between sepsis care started in the ED and the first alert triggering upon ward admission. ED indicates emergency department; RRT, rapid response team.

Figure 2.

Project life cycle of machine learning–based risk tools in acute care settings.