Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Meta-Analysis
. 2024 Jun 25;8(12):2991-3000.
doi: 10.1182/bloodadvances.2023012200.

Machine learning natural language processing for identifying venous thromboembolism: systematic review and meta-analysis

Barbara D Lam  1   2 , Pavlina Chrysafi  3 , Thita Chiasakul  4 , Harshit Khosla  5 , Dimitra Karagkouni  6 , Megan McNichol  7 , Alys Adamski  8 , Nimia Reyes  8 , Karon Abe  8 , Simon Mantha  9 , Ioannis S Vlachos  6 , Jeffrey I Zwicker  9 , Rushad Patell  1
Affiliations
Meta-Analysis

Machine learning natural language processing for identifying venous thromboembolism: systematic review and meta-analysis

Barbara D Lam et al. Blood Adv. .

Abstract

Venous thromboembolism (VTE) is a leading cause of preventable in-hospital mortality. Monitoring VTE cases is limited by the challenges of manual medical record review and diagnosis code interpretation. Natural language processing (NLP) can automate the process. Rule-based NLP methods are effective but time consuming. Machine learning (ML)-NLP methods present a promising solution. We conducted a systematic review and meta-analysis of studies published before May 2023 that use ML-NLP to identify VTE diagnoses in the electronic health records. Four reviewers screened all manuscripts, excluding studies that only used a rule-based method. A meta-analysis evaluated the pooled performance of each study's best performing model that evaluated for pulmonary embolism and/or deep vein thrombosis. Pooled sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) with confidence interval (CI) were calculated by DerSimonian and Laird method using a random-effects model. Study quality was assessed using an adapted TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) tool. Thirteen studies were included in the systematic review and 8 had data available for meta-analysis. Pooled sensitivity was 0.931 (95% CI, 0.881-0.962), specificity 0.984 (95% CI, 0.967-0.992), PPV 0.910 (95% CI, 0.865-0.941) and NPV 0.985 (95% CI, 0.977-0.990). All studies met at least 13 of the 21 NLP-modified TRIPOD items, demonstrating fair quality. The highest performing models used vectorization rather than bag-of-words and deep-learning techniques such as convolutional neural networks. There was significant heterogeneity in the studies, and only 4 validated their model on an external data set. Further standardization of ML studies can help progress this novel technology toward real-world implementation.

PubMed Disclaimer

Conflict of interest statement

Conflict-of-interest disclosure: R.P. reports consultancy with Merck. Outside current work, J.I.Z. reports prior research funding from Incyte and Quercegen; consultancy for Sanofi, CSL Behring, and Calyx. Outside current work, S.M. has served as an adviser for Janssen Pharmaceuticals and is the principal owner of Daboia Consulting LLC. S.M. has a patent application pending and is developing a licensing agreement with Superbio.ai for NLP software not featured in this work. T.C. reports consultancy for Takeda, outside of current work. The remaining authors declare no competing financial interests.

Figures

None
Graphical abstract
Figure 1.
Figure 1.
PRISMA diagram.
Figure 2.
Figure 2.
Performance assessment. (A) Pooled analysis of studies that reported sensitivity and specificity of venous thromboembolism identification by ML-based NLP. (B) Pooled analysis of studies that reported PPV and NPV of venous thromboembolism identification by ML-NLP.
Figure 3.
Figure 3.
Study adherence to TRIPOD guidelines modified for studies assessing NLP. Checklist items correspond to supplemental Table 1, the modified TRIPOD checklist for NLP studies.
See this image and copyright information in PMC

References

    1. Cohen AT, Tapson VF, Bergmann JF, et al. Venous thromboembolism risk and prophylaxis in the acute hospital care setting (ENDORSE study): a multinational cross-sectional study. Lancet. 2008;371(9610):387–394. - PubMed
    1. Fanikos J, Piazza G, Zayaruzny M, Goldhaber SZ. Long-term complications of medical patients with hospital-acquired venous thromboembolism. Thromb Haemost. 2009;102(4):688–693. - PubMed
    1. Henke PK, Kahn SR, Pannucci CJ, et al. Call to action to prevent venous thromboembolism in hospitalized patients: a policy statement from the American Heart Association. Circulation. 2020;141(24):e914–e931. - PubMed
    1. Agency of Healthcare Research and Quality Chapter 4. Choose the Model to Assess VTE and Bleeding Risk. https://www.ahrq.gov/patient-safety/settings/hospital/vtguide/guide4.html
    1. The Joint Commission Venous Thromboembolism. https://www.jointcommission.org/measurement/measures/venous-thromboembol...

Publication types