Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 May 1;184(5):557-562.
doi: 10.1001/jamainternmed.2024.0084.

Effectiveness of an Artificial Intelligence-Enabled Intervention for Detecting Clinical Deterioration

Robert J Gallo  1   2 , Lisa Shieh  3 , Margaret Smith  3   4 , Ben J Marafino  5 , Pascal Geldsetzer  3   6 , Steven M Asch  1   3 , Kenny Shum  3 , Steven Lin  3   4 , Jerri Westphal  3 , Grace Hong  3 , Ron Chen Li  3
Affiliations

Effectiveness of an Artificial Intelligence-Enabled Intervention for Detecting Clinical Deterioration

Robert J Gallo et al. JAMA Intern Med. .

Erratum in

  • Article Changed to Open Access.
    [No authors listed] [No authors listed] JAMA Intern Med. 2024 Sep 1;184(9):1137. doi: 10.1001/jamainternmed.2024.3693. JAMA Intern Med. 2024. PMID: 39037791 Free PMC article. No abstract available.

Abstract

Importance: Inpatient clinical deterioration is associated with substantial morbidity and mortality but may be easily missed by clinicians. Early warning scores have been developed to alert clinicians to patients at high risk of clinical deterioration, but there is limited evidence for their effectiveness.

Objective: To evaluate the effectiveness of an artificial intelligence deterioration model-enabled intervention to reduce the risk of escalations in care among hospitalized patients using a study design that facilitates stronger causal inference.

Design, setting, and participants: This cohort study used a regression discontinuity design that controlled for confounding and was based on Epic Deterioration Index (EDI; Epic Systems Corporation) prediction model scores. Compared with other observational research, the regression discontinuity design facilitates causal analysis. Hospitalized adults were included from 4 general internal medicine units in 1 academic hospital from January 17, 2021, through November 16, 2022.

Exposure: An artificial intelligence deterioration model-enabled intervention, consisting of alerts based on an EDI score threshold with an associated collaborative workflow among nurses and physicians.

Main outcomes and measures: The primary outcome was escalations in care, including rapid response team activation, transfer to the intensive care unit, or cardiopulmonary arrest during hospitalization.

Results: During the study, 9938 patients were admitted to 1 of the 4 units, with 963 patients (median [IQR] age, 76.1 [64.2-86.2] years; 498 males [52.3%]) included within the primary regression discontinuity analysis. The median (IQR) Elixhauser Comorbidity Index score in the primary analysis cohort was 10 (0-24). The intervention was associated with a -10.4-percentage point (95% CI, -20.1 to -0.8 percentage points; P = .03) absolute risk reduction in the primary outcome for patients at the EDI score threshold. There was no evidence of a discontinuity in measured confounders at the EDI score threshold.

Conclusions and relevance: Using a regression discontinuity design, this cohort study found that the implementation of an artificial intelligence deterioration model-enabled intervention was associated with a significantly decreased risk of escalations in care among inpatients. These results provide evidence for the effectiveness of this intervention and support its further expansion and testing in other care settings.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Ms Smith reported receiving grants from the American Board of Family Medicine Foundation, Google, and Omada Health outside the submitted work. Dr Gallo is supported by a VA Advanced Fellowship in Medical Informatics. No other disclosures were reported.

Figures

Figure 1.
Figure 1.. Proportion of Patients Who Triggered Alert Across Epic Deterioration Index (EDI) Scores
Patients were binned by 1.0-point increments in the EDI score above and below the threshold of 65.0 (vertical line).
Figure 2.
Figure 2.. Rate of Primary Outcome Across Epic Deterioration Index (EDI) Scores
Patients were binned by 1.0-point increments in the EDI score above and below the threshold of 65.0 (vertical line). The primary outcome was activation of the rapid response team, transfer to an intensive care unit, or cardiopulmonary arrest. Linear regression lines fit to the binned primary outcome rate above and below the EDI score threshold are shown.
See this image and copyright information in PMC

References

    1. Escobar GJ, Greene JD, Gardner MN, Marelich GP, Quick B, Kipnis P. Intra-hospital transfers to a higher level of care: contribution to total hospital and intensive care unit (ICU) mortality and length of stay (LOS). J Hosp Med. 2011;6(2):74-80. doi:10.1002/jhm.817 - DOI - PubMed
    1. Epic Systems . Artificial intelligence triggers fast, lifesaving care for COVID-19 patients. April 22, 2020. Accessed March 4, 2023. https://www.epic.com/epic/post/artificial-intelligence-epic-triggers-fas...
    1. Peelen RV, Eddahchouri Y, Koeneman M, van de Belt TH, van Goor H, Bredie SJ. Algorithms for prediction of clinical deterioration on the general wards: a scoping review. J Hosp Med. 2021;16(10):612-619. doi:10.12788/jhm.3630 - DOI - PubMed
    1. Kollef MH, Chen Y, Heard K, et al. . A randomized trial of real-time automated clinical deterioration alerts sent to a rapid response team. J Hosp Med. 2014;9(7):424-429. doi:10.1002/jhm.2193 - DOI - PMC - PubMed
    1. Escobar GJ, Liu VX, Schuler A, Lawson B, Greene JD, Kipnis P. Automated identification of adults at risk for in-hospital clinical deterioration. N Engl J Med. 2020;383(20):1951-1960. doi:10.1056/NEJMsa2001090 - DOI - PMC - PubMed

Publication types