Large Language Models for Psychiatric Phenotype Extraction from Electronic Health Records

Abstract

The accurate detection of clinical phenotypes from electronic health records (EHRs) is pivotal for advancing large-scale genetic and longitudinal studies in psychiatry. Free-text clinical notes are an essential source of symptom-level information, particularly in psychiatry. However, the automated extraction of symptoms from clinical text remains challenging. Here, we tested 11 open-source generative large language models (LLMs) for their ability to detect 109 psychiatric phenotypes from clinical text, using annotated EHR notes from a psychiatric clinic in Colombia. The LLMs were evaluated both "out-of-the-box" and after fine-tuning, and compared against a traditional natural language processing (tNLP) method developed from the same data. We show that while base LLM performance was poor to moderate (0.2-0.6 macro-F1 for zero-shot; 0.2-0.74 macro-F1 for few shot), it improved significantly after fine-tuning (0.75-0.86 macro-F1), with several fine-tuned LLMs outperforming the tNLP method. In total, 100 phenotypes could be reliably detected (F1>0.8) using either a fine-tuned LLM or tNLP. To generate a fine-tuned LLM that can be shared with the scientific and medical community, we created a fully synthetic dataset free of patient information but based on original annotations. We fine-tuned a top-performing LLM on this data, creating "Mistral-small-psych", an LLM that can detect psychiatric phenotypes from Spanish text with performance comparable to that of LLMs trained on real EHR data (macro-F1=0.79). Finally, the fine-tuned LLMs underwent an external validation using data from a large psychiatric hospital in Colombia, the Hospital Mental de Antioquia, highlighting that most LLMs generalized well (0.02-0.16 point loss in macro-F1). Our study underscores the value of domain-specific adaptation of LLMs and introduces a new model for accurate psychiatric phenotyping in Spanish text, paving the way for global precision psychiatry.

Figure 1:. Overview of extracted phenotypes and study design.

(a) Examples of psychiatric phenotypes extracted from EHR in this study (the full list can be found in Supplementary Table 1). (b) LLM application overview: LLMs were evaluated on the detection of 136 psychiatric phenotypes from EHR documents in zero-and few-shot settings (with the latter including up to five annotated examples of the phenotype of interest in the prompt); additionally, LLMs were fine-tuned using a training set of 1,477 annotated EHR documents.

Figure 2:. Mean performance (macro-F1) for all models

Mean LLM performance (macro-F1) on the CSJDM test set (n=358 documents), in zero-shot (light bars, left) and few-shot (medium bars, middle) settings, as well as after fine-tuning (dark bars, right) for all testable phenotypes (n=109 phenotypes). Error bars represent a standard deviation from the mean F1 score.

Figure 3:. Per-phenotype performance of all fine-tuned LLMs, against frequency in the training set.

Per-phenotype F1 for all LLMs fine-tuned on CSJDM data (colored lines); phenotypes are ordered by their frequency in the training data (bottom: gray bars showing the relative frequency of each phenotype in the CSJDM training set, n=1,477 documents). Highlighted phenotypes include examples of phenotypes with phenotype-specific (“other addictions”, “poor response to psychotropic medications”, “adverse effects”) and model-specific low performance (“poor adherence”), as well as high performance across-models despite low frequency in the training data (“hypothymia”).

Figure 4:. Overview of Mistral-small-psych’s creation and performance.

(a) Mistral-small-psych creation: Llama3-70B was prompted to generate synthetic clinical sentences containing published phenotype annotation examples extracted from real annotated EHR data. These synthetic sentences, which contained no patient information, were sampled and assembled into documents, further labeled using the tNLP system, and used as training data to fine-tune Mistral-small, creating Mistral-small-psych. (b) Phenotype-level performance (F1) of Mistral-small-psych (red), vs. all 11 LLMs evaluated in zero-shot (blue) and few-shot (purple) settings, with phenotypes ordered by Mistral-small-psych F1 score. (c) Performance and generalizability of all fine-tuned LLMs: in-domain (CSJDM, solid bars) vs. out-of-domain (HOMO, striped bars) macro-F1 scores across all fine-tuned models. Mistral-small-psych (right, red) was fine-tuned on synthetic data. For comparison purposes, results are presented for phenotypes with three or more instances in both the CSJDM and HOMO test sets (N phenotypes=97). Error bars represent a standard deviation from the mean F1 score.