ChatGPT-4 extraction of heart failure symptoms and signs from electronic health records

Abstract

Background: Natural language processing (NLP) can facilitate research utilizing data from electronic health records (EHRs). Large language models can potentially improve NLP applications leveraging EHR notes. The objective of this study was to assess the performance of zero-shot learning using Chat Generative Pre-trained Transformer 4 (ChatGPT-4) for extraction of symptoms and signs, and compare its performance to baseline machine learning and rule-based methods developed using annotated data.

Methods and results: From unstructured clinical notes of the national EHR data of the Veterans healthcare system, we extracted 1999 text snippets containing relevant keywords for heart failure symptoms and signs, which were then annotated by two clinicians. We also created 102 synthetic snippets that were semantically similar to snippets randomly selected from the original 1999 snippets. The authors applied zero-shot learning, using two different forms of prompt engineering in a symptom and sign extraction task with ChatGPT-4, utilizing the synthetic snippets. For comparison, baseline models using machine learning and rule-based methods were trained using the original 1999 annotated text snippets, and then used to classify the 102 synthetic snippets. The best zero-shot learning application achieved 90.6 % precision, 100 % recall, and 95 % F1 score, outperforming the best baseline method, which achieved 54.9 % precision, 82.4 % recall, and 65.5 % F1 score. Prompt style and temperature settings influenced zero-shot learning performance.

Conclusions: Zero-shot learning utilizing ChatGPT-4 significantly outperformed traditional machine learning and rule-based NLP. Prompt type and temperature settings affected zero-shot learning performance. These findings suggest a more efficient means of symptoms and signs extraction than traditional machine learning and rule-based methods.

Keywords: Heart failure; Information extraction; Large language models; Zero-shot learning.

Figure 1.

Annotation classification distributions by symptom/sign group

Figures 2a and 2b.

Classification without and with rule-based (RB) enhancement. In Figure 2a (top), all input synthetic snippets are classified by the trained machine learning (ML) model. In Figure 2b, for synthetic snippets that are classified by both the trained ML model and the RB enhancement, the RB classification is used in final output. Note that only synthetic snippets containing matching regular expression patterns can be classified by the RB enhancement.

Figure 3.

ZSL using the OpenAI API. Instructional prompt sent to OpenAI API, followed by the text to be analyzed; the API returns a “yes” or “no” answer indicating a positive or negative classification for the text.