Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
[Preprint]. 2023 Sep 11:2023.09.10.23295315.
doi: 10.1101/2023.09.10.23295315.

Automated Identification of Heart Failure with Reduced Ejection Fraction using Deep Learning-based Natural Language Processing

Arash A Nargesi  1   2 , Philip Adejumo  2 , Lovedeep Dhingra  2 , Benjamin Rosand  2 , Astrid Hengartner  2 , Andreas Coppi  2   3 , Simon Benigeri  4 , Sounok Sen  2 , Tariq Ahmad  2 , Girish N Nadkarni  5 , Zhenqiu Lin  3 , Faraz S Ahmad  4 , Harlan M Krumholz  2   3   6   7 , Rohan Khera  2   3
Affiliations

Automated Identification of Heart Failure with Reduced Ejection Fraction using Deep Learning-based Natural Language Processing

Arash A Nargesi et al. medRxiv. .

Update in

Abstract

Background: The lack of automated tools for measuring care quality has limited the implementation of a national program to assess and improve guideline-directed care in heart failure with reduced ejection fraction (HFrEF). A key challenge for constructing such a tool has been an accurate, accessible approach for identifying patients with HFrEF at hospital discharge, an opportunity to evaluate and improve the quality of care.

Methods: We developed a novel deep learning-based language model for identifying patients with HFrEF from discharge summaries using a semi-supervised learning framework. For this purpose, hospitalizations with heart failure at Yale New Haven Hospital (YNHH) between 2015 to 2019 were labeled as HFrEF if the left ventricular ejection fraction was under 40% on antecedent echocardiography. The model was internally validated with model-based net reclassification improvement (NRI) assessed against chart-based diagnosis codes. We externally validated the model on discharge summaries from hospitalizations with heart failure at Northwestern Medicine, community hospitals of Yale New Haven Health in Connecticut and Rhode Island, and the publicly accessible MIMIC-III database, confirmed with chart abstraction.

Results: A total of 13,251 notes from 5,392 unique individuals (mean age 73 ± 14 years, 48% female), including 2,487 patients with HFrEF (46.1%), were used for model development (train/held-out test: 70/30%). The deep learning model achieved an area under receiving operating characteristic (AUROC) of 0.97 and an area under precision-recall curve (AUPRC) of 0.97 in detecting HFrEF on the held-out set. In external validation, the model had high performance in identifying HFrEF from discharge summaries with AUROC 0.94 and AUPRC 0.91 on 19,242 notes from Northwestern Medicine, AUROC 0.95 and AUPRC 0.96 on 139 manually abstracted notes from Yale community hospitals, and AUROC 0.91 and AUPRC 0.92 on 146 manually reviewed notes at MIMIC-III. Model-based prediction of HFrEF corresponded to an overall NRI of 60.2 ± 1.9% compared with the chart diagnosis codes (p-value < 0.001) and an increase in AUROC from 0.61 [95% CI: 060-0.63] to 0.91 [95% CI 0.90-0.92].

Conclusions: We developed and externally validated a deep learning language model that automatically identifies HFrEF from clinical notes with high precision and accuracy, representing a key element in automating quality assessment and improvement for individuals with HFrEF.

Keywords: Deep learning; Electronic heart records; Heart failure with reduced ejection fraction; Longformer; Natural language processing.

PubMed Disclaimer

Conflict of interest statement

Disclosures Dr. Tariq Ahmad is a consultant for Sanofi-Aventis, Amgen, and Cytokinetics. He has research funding from Boehringer Ingelheim, AstraZeneca, Cytokinetics, and Relypsa. Dr. Nadkarni reports consultancy agreements with AstraZeneca, BioVie, GLG Consulting, Pensieve Health, Reata, Renalytix, Siemens Healthineers and Variant Bio; has received research funding from Goldfinch Bio, and Renalytix; has received honoraria from AstraZeneca, BioVie, Lexicon, Daiichi Sankyo, Meanrini Health, and Reata; has patents or royalties with Renalytix; owns equity and stock options in Pensieve Health and Renalytix as a scientific cofounder; owns equity in Verici Dx; has received financial compensation as a scientific board member and advisor to Renalytix; has served on the advisory board of Neurona Health; and has served in an advisory or leadership role for Pensieve Health and Renalytix. Dr. Faraz Ahmad reports grants from the Agency for Healthcare Research and Quality, grants from the National Institutes of Health/National Heart, Lung, and Blood Institute, grants from the American Heart Association, personal fees from Teladoc, personal fees from Livongo, personal fees from Pfizer, outside the submitted work. Dr. Krumholz works under contract with the Centers for Medicare & Medicaid Services to support quality measurement programs, was a recipient of a research grant from Johnson & Johnson, through Yale University, to support clinical trial data sharing; was a recipient of a research agreement, through Yale University, from the Shenzhen Center for Health Information for work to advance intelligent disease prevention and health promotion; collaborates with the National Center for Cardiovascular Diseases in Beijing; receives payment from the Arnold & Porter Law Firm for work related to the Sanofi clopidogrel litigation, from the Martin Baughman Law Firm for work related to the Cook Celect IVC filter litigation, and from the Siegfried and Jensen Law Firm for work related to Vioxx litigation; chairs a Cardiac Scientific Advisory Board for UnitedHealth; was a member of the IBM Watson Health Life Sciences Board; is a member of the Advisory Board for Element Science, the Advisory Board for Facebook, and the Physician Advisory Board for Aetna; and is the co-founder of Hugo Health, a personal health information platform, and co-founder of Refactor Health, a healthcare AI-augmented data management company. Dr. Khera is an Associate Editor of JAMA. He receives support from the National Heart, Lung, and Blood Institute of the National Institutes of Health (under award K23HL153775) and the Doris Duke Charitable Foundation (under award, 2022060). He also receives research support, through Yale, from Bristol-Myers Squibb and Novo Nordisk. He is a coinventor of U.S. Provisional Patent Applications 63/177,117, 63/428,569, 63/346,610, 63/484,426, and 3/508,315. He is a co-founder of Evidence2Health, a precision health platform. The remaining authors have no competing interests to disclose.

Figures

Figure 1.
Figure 1.. Model Development.
Hospitalizations for heart failure at Yale New Haven Hospital with an antecedent echocardiography were included for model development. The schematic represents data processing, model development, and evaluation of model performance on validation sets, including Northwestern Medicine, community hospitals of Yale New Haven health system, and MIMIC-III database. Abbreviations: YNHH, Yale New Haven Hospital; HFrEF, heart failure with reduced ejection fraction; MIMIC, Medical Information Mart for Intensive Care.
Figure 2.
Figure 2.. Model Performance.
The graphs represent receiver operating characteristic (left) and precision-recall (right) curves in identifying HFrEF on clinical notes from the held-out test set at Yale New Haven Hospital. Abbreviations: ROC, receiver operating characteristic curve; PRC, precision recall curve; AUC, area under curve.
Figure 3.
Figure 3.. External Validation.
Performance of the model in identifying HFrEF was evaluated on discharge summaries from three external sets, including 1) Northwestern Medicine, 2) Community hospital of Yale New Haven health system, and 3) MIMIC-III. The figure represents receiver operating characteristic (left) and precision-recall (right) curves for these analyses. Abbreviations: YNH, Yale New Haven; MIMIC, Medical Information Mart for Intensive Care III; AUC, area under curve.
Figure 4.
Figure 4.. External Validation.
The bar graph represents model performance in detecting HFrEF from discharge summaries of hospitalizations with heart failure at Northwestern Medicine (blue), community hospitals of Yale New Haven health (pink) and MIMIC-III (yellow). Abbreviations: AUROC, area under receiving operating characteristics; AUPRC, area under precision recall curve; PPV, positive predictive value; NPV, negative predictive value; MIMIC, Medical Information Mart for Intensive Care III.
Figure 5.
Figure 5.. Disease Classification and Use of Guideline Directed Therapies.
(A) Nested pie chart demonstrates disease classification based on echocardiography measurements of LVEF (true label, outer circle), model predictions of disease phenotype (middle circle), and chart-documented diagnosis codes (inner circle). Blue represents a diagnosis of HFrEF and red represents other phenotypes of heart failure; (B) Use of guideline-directed therapies among individuals with LVEF < 40% who were correctly reclassified from a chart-diagnosis of non-HFrEF to a model-based diagnosis of HFrEF. Abbreviations: ARNI, angiotensin receptor-neprilysin inhibitor; ARB, angiotensin receptor blocker; ACEI, angiotensin converting enzyme inhibitor; SGLT2i, sodium glucose cotransporter 2 inhibitor; MRA, mineralocorticoid receptor antagonist.
Figure 6.
Figure 6.. Predictive Keywords for Heart failure Phenotypes.
Figure represents Local Interpretable Model-agnostic Explanations (LIME) for model predictions of HFrEF from discharge summaries. (A) The most predictive keywords for HFrEF with corresponding coefficients based on the LIME analysis of the top 100 notes with the most confident model predictions. (B) A real-world example of LIME analysis on a discharge summary representing local dependencies. The color intensity of each highlighted keyword represents its contributions to the model-predicted probability of a HFrEF (orange) vs a non-HFrEF phenotype (blue). The deidentified discharge summary describes the hospitalization of an individual after initial presentation with heart failure and left ventricular ejection fraction of 30% on echocardiography. The model predicted a 99% probability for a HFrEF from this discharge summary. Abbreviations: MRI, magnetic resonance imaging; ICM, ischemic cardiomyopathy; VAD, ventricular assist device; NICM, non-ischemic cardiomyopathy; PAD, peripheral arterial disease; HFrEF, heart failure with reduced ejection fraction; PMH, past medical history; USH, usual state of health; ED, emergency department; HDS, hemodynamically stable; CXR, chest x-ray; ECHO, echocardiography; EF, ejection fraction; BNP, B-natriuretic peptide.
See this image and copyright information in PMC

References

    1. Benjamin EJ, Muntner P, Alonso A et al. Heart Disease and Stroke Statistics-2019 Update: A Report From the American Heart Association. Circulation 2019;139:e56–e528. - PubMed
    1. Tromp J, Ouwerkerk W, van Veldhuisen DJ et al. A Systematic Review and Network Meta-Analysis of Pharmacological Treatment of Heart Failure With Reduced Ejection Fraction. JACC Heart Fail 2022;10:73–84. - PubMed
    1. Roth GA, Poole JE, Zaha R, Zhou W, Skinner J, Morden NE. Use of Guideline-Directed Medications for Heart Failure Before Cardioverter-Defibrillator Implantation. J Am Coll Cardiol 2016;67:1062–1069. - PMC - PubMed
    1. Greene SJ, Butler J, Albert NM et al. Medical Therapy for Heart Failure With Reduced Ejection Fraction: The CHAMP-HF Registry. J Am Coll Cardiol 2018;72:351–366. - PubMed
    1. Kreimeyer K, Foster M, Pandey A et al. Natural language processing systems for capturing and standardizing unstructured clinical information: A systematic review. J Biomed Inform 2017;73:14–29. - PMC - PubMed

Publication types