Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Oct 1;7(10):e2438986.
doi: 10.1001/jamanetworkopen.2024.38986.

Early Warning Scores With and Without Artificial Intelligence

Dana P Edelson  1   2 , Matthew M Churpek  3 , Kyle A Carey  1 , Zhenqui Lin  4 , Chenxi Huang  4 , Jonathan M Siner  5 , Jennifer Johnson  6 , Harlan M Krumholz  4 , Deborah J Rhodes  7
Affiliations

Early Warning Scores With and Without Artificial Intelligence

Dana P Edelson et al. JAMA Netw Open. .

Erratum in

  • Error in Author Name.
    [No authors listed] [No authors listed] JAMA Netw Open. 2024 Nov 4;7(11):e2448969. doi: 10.1001/jamanetworkopen.2024.48969. JAMA Netw Open. 2024. PMID: 39509136 Free PMC article. No abstract available.

Abstract

Importance: Early warning decision support tools to identify clinical deterioration in the hospital are widely used, but there is little information on their comparative performance.

Objective: To compare 3 proprietary artificial intelligence (AI) early warning scores and 3 publicly available simple aggregated weighted scores.

Design, setting, and participants: This retrospective cohort study was performed at 7 hospitals in the Yale New Haven Health System. All consecutive adult medical-surgical ward hospital encounters between March 9, 2019, and November 9, 2023, were included.

Exposures: Simultaneous Epic Deterioration Index (EDI), Rothman Index (RI), eCARTv5 (eCART), Modified Early Warning Score (MEWS), National Early Warning Score (NEWS), and NEWS2 scores.

Main outcomes and measures: Clinical deterioration, defined as a transfer from ward to intensive care unit or death within 24 hours of an observation.

Results: Of the 362 926 patient encounters (median patient age, 64 [IQR, 47-77] years; 200 642 [55.3%] female), 16 693 (4.6%) experienced a clinical deterioration event. eCART had the highest area under the receiver operating characteristic curve at 0.895 (95% CI, 0.891-0.900), followed by NEWS2 at 0.831 (95% CI, 0.826-0.836), NEWS at 0.829 (95% CI, 0.824-0.835), RI at 0.828 (95% CI, 0.823-0.834), EDI at 0.808 (95% CI, 0.802-0.812), and MEWS at 0.757 (95% CI, 0.750-0.764). After matching scores at the moderate-risk sensitivity level for a NEWS score of 5, overall positive predictive values (PPVs) ranged from a low of 6.3% (95% CI, 6.1%-6.4%) for an EDI score of 41 to a high of 17.3% (95% CI, 16.9%-17.8%) for an eCART score of 94. Matching scores at the high-risk specificity of a NEWS score of 7 yielded overall PPVs ranging from a low of 14.5% (95% CI, 14.0%-15.2%) for an EDI score of 54 to a high of 23.3% (95% CI, 22.7%-24.2%) for an eCART score of 97. The moderate-risk thresholds provided a median of at least 20 hours of lead time for all the scores. Median lead time at the high-risk threshold was 11 (IQR, 0-69) hours for eCART, 8 (IQR, 0-63) hours for NEWS, 6 (IQR, 0-62) hours for NEWS2, 5 (IQR, 0-56) hours for MEWS, 1 (IQR, 0-39) hour for EDI, and 0 (IQR, 0-42) hours for RI.

Conclusions and relevance: In this cohort study of inpatient encounters, eCART outperformed the other AI and non-AI scores, identifying more deteriorating patients with fewer false alarms and sufficient time to intervene. NEWS, a non-AI, publicly available early warning score, significantly outperformed EDI. Given the wide variation in accuracy, additional transparency and oversight of early warning tools may be warranted.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Dr Edelson reported equity interest in AgileMD and having a patent for eCART with royalites paid by University of Chicago outside the submitted work. Dr Churpek reported having patent for eCART with royalties paid by University of Chicago. No other disclosures were reported.

Figures

Figure.
Figure.. Observation-Level Precision Recall Curve and Encounter-Level Efficiency Curve for Each Score
Precision is measured as positive predictive value (PPV); recall, as sensitivity. eCART indicates eCARTv5; EDI, Epic Deterioration Index; MEWS, Modified Early Warning Score; NEWS, National Early Warning Score; and RI, Rothman Index.
See this image and copyright information in PMC

Comment in

References

    1. Churpek MM, Wendlandt B, Zadravecz FJ, Adhikari R, Winslow C, Edelson DP. Association between intensive care unit transfer delay and hospital mortality: a multicenter investigation. J Hosp Med. 2016;11(11):757-762. doi:10.1002/jhm.2630 - DOI - PMC - PubMed
    1. Barwise A, Thongprayoon C, Gajic O, Jensen J, Herasevich V, Pickering BW. Delayed rapid response team activation is associated with increased hospital mortality, morbidity, and length of stay in a tertiary care institution. Crit Care Med. 2016;44(1):54-63. doi:10.1097/CCM.0000000000001346 - DOI - PubMed
    1. Gupta S, Green C, Subramaniam A, Zhen LD, Low E, Tiruvoipati R. The impact of delayed rapid response call activation on patient outcomes. J Crit Care. 2017;41:86-90. doi:10.1016/j.jcrc.2017.05.006 - DOI - PubMed
    1. Young MP, Gooder VJ, McBride K, James B, Fisher ES. Inpatient transfers to the intensive care unit: delays are associated with increased mortality and morbidity. J Gen Intern Med. 2003;18(2):77-83. doi:10.1046/j.1525-1497.2003.20441.x - DOI - PMC - PubMed
    1. Gerry S, Bonnici T, Birks J, et al. . Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology. BMJ. 2020;369:m1501. doi:10.1136/bmj.m1501 - DOI - PMC - PubMed

Publication types