Mitigating the risk of health inequity exacerbated by large language models

Abstract

Recent advancements in large language models (LLMs) have demonstrated their potential in numerous medical applications, particularly in automating clinical trial matching for translational research and enhancing medical question-answering for clinical decision support. However, our study shows that incorporating non-decisive socio-demographic factors, such as race, sex, income level, LGBT+ status, homelessness, illiteracy, disability, and unemployment, into the input of LLMs can lead to incorrect and harmful outputs. These discrepancies could worsen existing health disparities if LLMs are broadly implemented in healthcare. To address this issue, we introduce EquityGuard, a novel framework designed to detect and mitigate the risk of health inequities in LLM-based medical applications. Our evaluation demonstrates its effectiveness in promoting equitable outcomes across diverse populations.

Fig. 1. Inequities when applying LLMs to two major medical applications.

Clinical Trial Matching (left) and Medical Question Answering (right). On the left, including race and sex information (e.g., “African-American” and “woman”) in the patient note, despite being irrelevant to matching the correct clinical trials, resulted in altered clinical trial recommendations generated by the LLMs. On the right, adding race information (e.g., “Native American”) to the question, which should not affect the response, led to incorrect answers from the LLMs. These examples show that non-decisive socio-demographic factors in different patient populations can lead to incorrect outputs from LLMs, which may lead to harmful clinical outcomes to these patient populations and eventually exacerbate healthcare inequities.

Fig. 2. Performance of various LLMs when specific SDOH factors were introduced into the dataset.

The clinical trial matching (CTM) performance is measured using NDCG@10 (higher is better), while the medical question answering (MQA) performance is measured using error rate (lower is better). SDOH factors include race, sex, low income, LGBT+ status, homelessness, illiteracy, disability, and unemployment. Each sensitive attribute was incorporated into the input data for both CTM and MQA tasks during the evaluation.

Fig. 3. Fairness metrics including equal opportunity (EO) and demographic parity (DP) to assess equity in LLMs.

Higher EO and DP scores indicate better equity, with EO focusing on ensuring equal positive outcomes for qualified individuals across groups and DP evaluating overall equity across all groups.

Fig. 4. Correlation heatmaps of inequity categories in CTM and MQA tasks.

The left plot shows the correlation between inequity categories in CTM tasks, illustrating how different inequity-modified queries resulted in similar trial rankings or selections by the models. The right plot shows the correlation between inequity categories in MQA tasks, displaying how often different inequity-modified queries led to the same answers or error patterns. These heatmaps help analyze how inequities across categories are interconnected, impacting model fairness.

Fig. 5. Equal opportunity (EO) and demographic parity (DP) metrics for LLaMA3 8B models.

Models trained with EquityGuard (`w/EquityGuard̀) show reduced EO and DP differences, indicating enhanced fairness.

Fig. 6

An overview of the EquityGuard framework for inequity detection and correction.