A Large Language Model-Based Generative Natural Language Processing Framework Finetuned on Clinical Notes Accurately Extracts Headache Frequency from Electronic Health Records

Abstract

Background: Headache frequency, defined as the number of days with any headache in a month (or four weeks), remains a key parameter in the evaluation of treatment response to migraine preventive medications. However, due to the variations and inconsistencies in documentation by clinicians, significant challenges exist to accurately extract headache frequency from the electronic health record (EHR) by traditional natural language processing (NLP) algorithms.

Methods: This was a retrospective cross-sectional study with human subjects identified from three tertiary headache referral centers- Mayo Clinic Arizona, Florida, and Rochester. All neurology consultation notes written by more than 10 headache specialists between 2012 to 2022 were extracted and 1915 notes were used for model fine-tuning (90%) and testing (10%). We employed four different NLP frameworks: (1) ClinicalBERT (Bidirectional Encoder Representations from Transformers) regression model (2) Generative Pre-Trained Transformer-2 (GPT-2) Question Answering (QA) Model zero-shot (3) GPT-2 QA model few-shot training fine-tuned on Mayo Clinic notes; and (4) GPT-2 generative model few-shot training fine-tuned on Mayo Clinic notes to generate the answer by considering the context of included text.

Results: The GPT-2 generative model was the best-performing model with an accuracy of 0.92[0.91 - 0.93] and R² score of 0.89[0.87, 0.9], and all GPT2-based models outperformed the ClinicalBERT model in terms of the exact matching accuracy. Although the ClinicalBERT regression model had the lowest accuracy 0.27[0.26 - 0.28], it demonstrated a high R² score 0.88[0.85, 0.89], suggesting the ClinicalBERT model can reasonably predict the headache frequency within a range of ≤ ± 3 days, and the R² score was higher than the GPT-2 QA zero-shot model or GPT-2 QA model few-shot training fine-tuned model.

Conclusion: We developed a robust model based on a state-of-the-art large language model (LLM)- a GPT-2 generative model that can extract headache frequency from EHR free-text clinical notes with high accuracy and R² score. It overcame several challenges related to different ways clinicians document headache frequency that were not easily achieved by traditional NLP models. We also showed that GPT2-based frameworks outperformed ClinicalBERT in terms of accuracy in extracting headache frequency from clinical notes. To facilitate research in the field, we released the GPT-2 generative model and inference code with open-source license of community use in GitHub.

Keywords: artificial intelligence; headache frequency; large language model; migraine; natural language processing.

Fig 1.. Nature Language Processing (NLP) framework for extracting headache frequency from neurology consultation notes - four parallel modeling schemes:

(1) ClinicalBERT regression model: encoder-based regression model pre-trained on MIMIC-III; (2) GPT-2 QA model zero-shot: decoder QA model trained on the generic web scraped data; (3) GPT-2 QA model few-shot training: decoder model trained on the generic web scraped data and fine-tuned on Mayo Clinic notes; and (4) GPT-2 generative model few-shot training: decoder model trained on the generic web scraped data and fine-tuned on Mayo Clinic notes to generate directly the answer by considering the context text. Abbreviations: Name Entity Recognition (NER); The Medical Information Mart for Intensive Care III (MIMIC-III); Generative Pre-Trained Transformer-2 (GPT-2); Question Answering (QA)

Fig 2.. Scatter plot visualization of true frequency reported in clinic notes (x-axis) and predicted frequency by the model (y-axis) in log scale.

Blue line shows the perfect alignment (R² = 1.0).

Fig 3.. Bland-Altman test to compare each metric computed from the NLP model against the ground truth.

There are 192 data elements in total for each subfigure, with each point representing one note in the validation dataset. Mean and standard deviation is also calculated. *The (b) GPT-2 QA zero-shot model incorrectly predicted 550 days for a sentence that includes “naproxen 550 mg”. Therefore, the range of prediction (X-axis) is much larger than other models.