External Validation of Deep Learning-Based Cardiac Arrest Risk Management System for Predicting In-Hospital Cardiac Arrest in Patients Admitted to General Wards Based on Rapid Response System Operating and Nonoperating Periods: A Single-Center Study

Abstract

Objectives: The limitations of current early warning scores have prompted the development of deep learning-based systems, such as deep learning-based cardiac arrest risk management systems (DeepCARS). Unfortunately, in South Korea, only two institutions operate 24-hour Rapid Response System (RRS), whereas most hospitals have part-time or no RRS coverage at all. This study validated the predictive performance of DeepCARS during RRS operation and nonoperation periods and explored its potential beyond RRS operating hours.

Design: Retrospective cohort study.

Setting: In this 1-year retrospective study conducted at Yonsei University Health System Severance Hospital in South Korea, DeepCARS was compared with conventional early warning systems for predicting in-hospital cardiac arrest (IHCA). The study focused on adult patients admitted to the general ward, with the primary outcome being IHCA-prediction performance within 24 hours of the alarm.

Patients: We analyzed the data records of adult patients admitted to a general ward from September 1, 2019, to August 31, 2020.

Interventions: None.

Measurements and main results: Performance evaluation was conducted separately for the operational and nonoperational periods of the RRS, using the area under the receiver operating characteristic curve (AUROC) as the metric. DeepCARS demonstrated a superior AUROC as compared with the Modified Early Warning Score (MEWS) and the National Early Warning Score (NEWS), both during RRS operating and nonoperating hours. Although the MEWS and NEWS exhibited varying performance across the two periods, DeepCARS showed consistent performance.

Conclusions: The accuracy and efficiency for predicting IHCA of DeepCARS were superior to that of conventional methods, regardless of whether the RRS was in operation. These findings emphasize that DeepCARS is an effective screening tool suitable for hospitals with full-time RRS, part-time RRS, and even those without any RRS.

Figure 1.

Comparison of predictive performance of deep learning-based cardiac arrest risk management system (DeepCARS) and the conventional Early Warning Score both during Rapid Response System (RRS) operating and nonoperating hours. AUROC = area under the receiver operating characteristic curve, AUPRC = area under the precision–recall curve, MEWS = Modified Early Warning Score, NEWS = National Early Warning Score, Sen = sensitivity, Spec =specificity, SPTTS = Single-Parameter Track-and-Trigger System.

Figure 2.

Precision–recall graph of deep learning-based cardiac arrest risk management system (DeepCARS) and Conventional Early Warning Scores. The solid line indicates the positive-predictive value. The dashed line indicates the F-measure. PPV = Positive-Predictive Value, MEWS = Modified Early Warning Score, NEWS = National Early Warning Score, RRS = Rapid Response System, SPTTS = Single-Parameter Track-and-Trigger System.

Figure 3.

Predictive performance based on varying prediction time. AUROC = area under the receiver operating characteristic curve, DeepCARS = deep learning-based cardiac arrest risk management system, MEWS = Modified Early Warning Score, NEWS = National Early Warning Score, RRS = Rapid Response System.

Figure 4.

Comparison of the alerting performance (alarms) of deep learning-based cardiac arrest risk management system (DeepCARS) and conventional early warning system both during Rapid Response System (RRS) operating and nonoperating periods. MEWS = Modified Early Warning Score, NEWS = National Early Warning Score, SPTTS = Single-Parameter Track-and-Trigger System.

Figure 5.

F-measure and positive-predictive value graph based on varying numbers of subsequent alarms. PPV = positive-predictive value, RRS = Rapid Response System.