Using Supervised Machine Learning to Classify Real Alerts and Artifact in Online Multisignal Vital Sign Monitoring Data

Abstract

Objective: The use of machine-learning algorithms to classify alerts as real or artifacts in online noninvasive vital sign data streams to reduce alarm fatigue and missed true instability.

Design: Observational cohort study.

Setting: Twenty-four-bed trauma step-down unit.

Patients: Two thousand one hundred fifty-three patients.

Intervention: Noninvasive vital sign monitoring data (heart rate, respiratory rate, peripheral oximetry) recorded on all admissions at 1/20 Hz, and noninvasive blood pressure less frequently, and partitioned data into training/validation (294 admissions; 22,980 monitoring hours) and test sets (2,057 admissions; 156,177 monitoring hours). Alerts were vital sign deviations beyond stability thresholds. A four-member expert committee annotated a subset of alerts (576 in training/validation set, 397 in test set) as real or artifact selected by active learning, upon which we trained machine-learning algorithms. The best model was evaluated on test set alerts to enact online alert classification over time.

Measurements and main results: The Random Forest model discriminated between real and artifact as the alerts evolved online in the test set with area under the curve performance of 0.79 (95% CI, 0.67-0.93) for peripheral oximetry at the instant the vital sign first crossed threshold and increased to 0.87 (95% CI, 0.71-0.95) at 3 minutes into the alerting period. Blood pressure area under the curve started at 0.77 (95% CI, 0.64-0.95) and increased to 0.87 (95% CI, 0.71-0.98), whereas respiratory rate area under the curve started at 0.85 (95% CI, 0.77-0.95) and increased to 0.97 (95% CI, 0.94-1.00). Heart rate alerts were too few for model development.

Conclusions: Machine-learning models can discern clinically relevant peripheral oximetry, blood pressure, and respiratory rate alerts from artifacts in an online monitoring dataset (area under the curve > 0.87).

Figure 1

Proportion of vital sign alert events (VSAE) lasting >1 minute, 1 to 2 minutes, 2-3 minutes and >3 minutes, in terms of count and aggregated length, for both data sets combined for vitals other than BP, as described in the text.

Figure 2

Timeframes of the first experiment (Panel A) for offline classification of alerts, and for the second experiment (Panel B) for online classification as the vital sign alert events (VSAE) evolve. A. In Experiment 1, which classifies alerts as either real or artifact offline, only the initial period of VSAE is considered: from the time the vital sign first exceeds the threshold of abnormality, until 3 minutes into the event. B. In the Experiment 2, which classifies alerts as either real or artifact online (as they evolve), a sequence of 3 minute windows is considered. The first of these windows ends at the start of VSAE, the last of them ends at 3 minute mark into VSAE, and the sequence is spaced at 20 second intervals.

Figure 3

Area Under the Receiver Operating Charcteristic Curve (AUC) scores charted for each of the time windows for time elapsed since alert start (i.e. since a vital sign (VS)crossed its threshold). Results for the 3 minute window ending at the alert start correspond to the time of 0 seconds in the graphs. Results for the window ending 180 seconds into the alert corrrespond 180s time marks. Panel A shows the scores from 10-fold cross validation on the training/validation set; the shaded bands represent the 95%-ile bootstrap confidence intervals computed using 1,000 random splits of training/validation data. Panel B shows the results on the external test set given the model learned from the 3-minute epochs of training/validation data; the shaded bands represents the 95%-ile bootrstrap confidence intervals computed using 1,000 draws of the held-out test data. For all three VS, artifacts were well discriminated from real alerts event at the alert start, suggesting that artifact and real alerts are discernible even in advance of the the VS reaching the threshhold of instability. Respiratory rate (RR) shows a more progressive trend in which the discrimiative power increases further from about 1.5 minutes into the alert, as compared to milder trends dispayed for blood pressure (BP) and pulse oximetry (SpO₂). A. Online performance observed using 10-fold cross validation on train/validation set B. Online performance observed with the external test set

Figure 4

Average prediction scores for groups of alerts annotated as real (solid line) and as artifacts (dotted line), charted along the time axis with respect to the alert time elapsed, obtained from the external test set using the model trained on training/validation set. The gray bands show 95%-ile confidence intervals. Discernability of true alerts vs. artifact increases as more vital sign information is collected over duration of vital sign alert event (VSAE) episodes. Key: HR=heart rate, RR=respiratory rate, SpO₂=pulse oximetry, BP=blood pressure.