Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Meta-Analysis
. 2024 Jul;31(7):696-706.
doi: 10.1111/acem.14937. Epub 2024 May 16.

Using natural language processing in emergency medicine health service research: A systematic review and meta-analysis

Hao Wang  1 , Naomi Alanis  1 , Laura Haygood  2 , Thomas K Swoboda  3 , Nathan Hoot  1 , Daniel Phillips  1 , Heidi Knowles  1 , Sara Ann Stinson  4 , Prachi Mehta  1 , Usha Sambamoorthi  5
Affiliations
Meta-Analysis

Using natural language processing in emergency medicine health service research: A systematic review and meta-analysis

Hao Wang et al. Acad Emerg Med. 2024 Jul.

Abstract

Objectives: Natural language processing (NLP) represents one of the adjunct technologies within artificial intelligence and machine learning, creating structure out of unstructured data. This study aims to assess the performance of employing NLP to identify and categorize unstructured data within the emergency medicine (EM) setting.

Methods: We systematically searched publications related to EM research and NLP across databases including MEDLINE, Embase, Scopus, CENTRAL, and ProQuest Dissertations & Theses Global. Independent reviewers screened, reviewed, and evaluated article quality and bias. NLP usage was categorized into syndromic surveillance, radiologic interpretation, and identification of specific diseases/events/syndromes, with respective sensitivity analysis reported. Performance metrics for NLP usage were calculated and the overall area under the summary of receiver operating characteristic curve (SROC) was determined.

Results: A total of 27 studies underwent meta-analysis. Findings indicated an overall mean sensitivity (recall) of 82%-87%, specificity of 95%, with the area under the SROC at 0.96 (95% CI 0.94-0.98). Optimal performance using NLP was observed in radiologic interpretation, demonstrating an overall mean sensitivity of 93% and specificity of 96%.

Conclusions: Our analysis revealed a generally favorable performance accuracy in using NLP within EM research, particularly in the realm of radiologic interpretation. Consequently, we advocate for the adoption of NLP-based research to augment EM health care management.

Keywords: electronic health records; emergency medicine; natural language processing; performance accuracy; precision; recall.

PubMed Disclaimer

Figures

Figure 1.
Figure 1.
PRISMA Flow Diagram: Using NLP in Emergency Medicine Research
Figure 2.
Figure 2.
Performance (Sensitivity and Specificity) of using NLP in Emergency Medicine Research
Figure 3.
Figure 3.
A Summary Receiver Operating Characteristic Curve
Figure 4.
Figure 4.
Fagan Plot
See this image and copyright information in PMC

References

    1. Obermeyer Z, Emanuel EJ. Predicting the Future - Big Data, Machine Learning, and Clinical Medicine. N Engl J Med 2016; 375:1216–1219. - PMC - PubMed
    1. Rajkomar A, Dean J, Kohane I. Machine Learning in Medicine. N Engl J Med 2019; 380:1347–1358. - PubMed
    1. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017; 542:115–118. - PMC - PubMed
    1. Miotto R, Li L, Kidd BA, Dudley JT. Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records. Sci Rep 2016; 6:26094. - PMC - PubMed
    1. Nemati S, Holder A, Razmi F, Stanley MD, Clifford GD, Buchman TG. An Interpretable Machine Learning Model for Accurate Prediction of Sepsis in the ICU. Crit Care Med 2018; 46:547–553. - PMC - PubMed

Publication types

LinkOut - more resources