Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models

Abstract

Many areas of medicine would benefit from deeper, more accurate phenotyping, but there are limited approaches for phenotyping using clinical notes without substantial annotated data. Large language models (LLMs) have demonstrated immense potential to adapt to novel tasks with no additional training by specifying task-specific instructions. Here we report the performance of a publicly available LLM, Flan-T5, in phenotyping patients with postpartum hemorrhage (PPH) using discharge notes from electronic health records (n = 271,081). The language model achieves strong performance in extracting 24 granular concepts associated with PPH. Identifying these granular concepts accurately allows the development of interpretable, complex phenotypes and subtypes. The Flan-T5 model achieves high fidelity in phenotyping PPH (positive predictive value of 0.95), identifying 47% more patients with this complication compared to the current standard of using claims codes. This LLM pipeline can be used reliably for subtyping PPH and outperforms a claims-based approach on the three most common PPH subtypes associated with uterine atony, abnormal placentation, and obstetric trauma. The advantage of this approach to subtyping is its interpretability, as each concept contributing to the subtype determination can be evaluated. Moreover, as definitions may change over time due to new guidelines, using granular concepts to create complex phenotypes enables prompt and efficient updating of the algorithm. Using this language modelling approach enables rapid phenotyping without the need for any manually annotated training data across multiple clinical use cases.

Fig. 1. Obstetric delivery cohort.

a Inclusion criteria for identifying delivery-related discharge summaries. The final cohort consists of 271,081 discharge notes for 131,284 women with an obstetric encounter at Mass General Brigham hospitals from 1998–2015. b–e Characteristics of the cohort based on (b) delivery hospital and (c) patient age at delivery, (d) patient race, and (e) patient ethnicity. ER emergency room, MGH Massachusetts General Hospital, BWH Brigham and Women’s Hospital, NSM North Shore Medical Center, NWH Newton Wellesley Hospital.

Fig. 2. Overview of zero-shot NLP pipeline for accurate and interpretable postpartum hemorrhage (PPH) phenotyping.

a Zero-shot extraction of PPH concepts using Flan-T5. We constructed either yes/no or extraction prompts for each PPH concept, concatenated the discharge summary of interest, and fed the combined input into Flan-T5. This process enabled the rapid extraction of PPH-related information from notes with no training data. b We leveraged the extracted concepts to perform interpretable phenotyping of PPH defined as cesarean deliveries with at least 1L or vaginal deliveries with at least 500mL of estimated blood loss. c We used the extracted concepts to perform interpretable subtyping of PPH based on the underlying etiology. We assigned a delivery note to the “tone", “tissue", “trauma", or “thrombin" subtype if any of the concepts associated with the subtype are present. PPH postpartum hemorrhage, EBL estimated blood loss, image: Flaticon.com.

Fig. 3. Comparison of the Flan-T5 language model and regular expression approaches for zero-shot extraction of postpartum hemorrhage (PPH)-related concepts.

The prevalence of each concept in the annotated test set is reported and compared to the model performance according to binary F1 score. The stars (*) denote that there is a significant difference (p < 0.05, McNemar test) between the regex and language model performance. PPH postpartum hemorrhage.

Fig. 4. Prevalence of postpartum hemorrhage (PPH) international classification of diseases (ICD) codes in notes with varying estimated blood loss (EBL).

The y-axis depicts the range of estimated blood loss (EBL) values extracted from the delivery note by Flan-T5 (if any), and the x-axis denotes the proportion of notes in each EBL category with a postpartum hemorrhage ICD diagnostic code. PPH ICD codes are defined according to the definition in Zheutlin et al.. Refer to Supplementary Fig. 2 for a similar plot using the PPH ICD definition from Butwick et al.. PPH postpartum hemorrhage, ICD international classification of diseases, EBL estimated blood loss.