Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Oct 14;8(10):1677.
doi: 10.3390/jcm8101677.

Artificial Intelligence Algorithms and Natural Language Processing for the Recognition of Syncope Patients on Emergency Department Medical Records

Franca Dipaola  1   2 , Mauro Gatti  3 , Veronica Pacetti  4 , Anna Giulia Bottaccioli  5 , Dana Shiffer  6   7 , Maura Minonzio  8   9 , Roberto Menè  10 , Alessandro Giaj Levra  11 , Monica Solbiati  12 , Giorgio Costantino  13 , Marco Anastasio  14 , Elena Sini  15 , Franca Barbic  16   17 , Enrico Brunetta  18   19 , Raffaello Furlan  20   21
Affiliations

Artificial Intelligence Algorithms and Natural Language Processing for the Recognition of Syncope Patients on Emergency Department Medical Records

Franca Dipaola et al. J Clin Med. .

Abstract

Background: Enrollment of large cohorts of syncope patients from administrative data is crucial for proper risk stratification but is limited by the enormous amount of time required for manual revision of medical records.

Aim: To develop a Natural Language Processing (NLP) algorithm to automatically identify syncope from Emergency Department (ED) electronic medical records (EMRs).

Methods: De-identified EMRs of all consecutive patients evaluated at Humanitas Research Hospital ED from 1 December 2013 to 31 March 2014 and from 1 December 2015 to 31 March 2016 were manually annotated to identify syncope. Records were combined in a single dataset and classified. The performance of combined multiple NLP feature selectors and classifiers was tested. Primary Outcomes: NLP algorithms' accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and F3 score.

Results: 15,098 and 15,222 records from 2013 and 2015 datasets were analyzed. Syncope was present in 571 records. Normalized Gini Index feature selector combined with Support Vector Machines classifier obtained the best F3 value (84.0%), with 92.2% sensitivity and 47.4% positive predictive value. A 96% analysis time reduction was computed, compared with EMRs manual review.

Conclusions: This artificial intelligence algorithm enabled the automatic identification of a large population of syncope patients using EMRs.

Keywords: Emergency Department; artificial intelligence; electronic medical records; natural language processing; syncope.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
Data analysis methodology. The flow chart summarizes the data analysis methodology. The 2013 and 2015 datasets were obtained from the Humanitas Research Hospital electronic repository. Data were manually annotated by a group of six physicians. Manual annotation is the process of reviewing medical records to identify patients with syncope. Each dataset comprised 15,098 and 15,222 electronic medical records, respectively. After filtering for inclusion (i.e., age ≥ 18 years and data completeness) and exclusion (i.e., age < 18 years or data incompleteness) criteria, the two datasets were combined in a single study dataset which underwent a preprocessing based on cleaning and formatting operations. The final study dataset was analyzed using a five-folds Nested Cross Validation (NCV). This latter was based on multiple iterations in each of which data was randomly split in five folds, i.e., five subsets of electronic medical records of equal magnitude. Four folds were used for feature selection/training/validation, whereas fold five was used for testing. Feature selection recognized the grams that were most relevant for identifying patients with syncope. Classifier training identified the model parameters that best represented the dataset. Validation enabled the identification of the best hyper-parameters model. NCV allowed the use of the entire set of data for either the training, validation or testing procedures, thus achieving algorithm optimal performance.
See this image and copyright information in PMC

References

    1. Costantino G., Perego F., Dipaola F., Borella M., Galli A., Cantoni G., Dell’Orto S., Dassi S., Filardo N., Duca P.G., et al. Short- and long-term prognosis of syncope, risk factors, and role of hospital admission: Results from the STePS (Short-Term Prognosis of Syncope) study. J. Am. Coll. Cardiol. 2008;51:276–283. doi: 10.1016/j.jacc.2007.08.059. - DOI - PubMed
    1. Numeroso F., Mossini G., Lippi G., Cervellin G. Analysis of Temporal and Causal Relationship Between Syncope and 30-Day Events in a Cohort of Emergency Department Patients to Identify the True Rate of Short-term Outcomes. J. Emerg. Med. 2018;55:612–619. doi: 10.1016/j.jemermed.2018.07.028. - DOI - PubMed
    1. Costantino G., Casazza G., Reed M., Bossi I., Sun B., Del Rosso A., Ungar A., Grossman S., D’Ascenzo F., Quinn J., et al. Syncope Risk Stratification Tools vs. Clinical Judgment: An Individual Patient Data Meta-analysis. Am. J. Med. 2014;127:1126.e13–1126.e25. doi: 10.1016/j.amjmed.2014.05.022. - DOI - PubMed
    1. Quinn J.V., Stiell I.G., A McDermott D., Sellers K.L., A Kohn M., A Wells G. Derivation of the San Francisco Syncope Rule to predict patients with short-term serious outcomes. Ann. Emerg. Med. 2004;43:224–232. doi: 10.1016/S0196-0644(03)00823-0. - DOI - PubMed
    1. Colivicchi F., Ammirati F., Melina D., Guido V., Imperoli G., Santini M. Development and prospective validation of a risk stratification system for patients with syncope in the emergency department: The OESIL risk score. ACC Curr. J. Rev. 2003;12:70–71. doi: 10.1016/j.accreview.2003.08.093. - DOI - PubMed