Social determinants of health extraction from clinical notes across institutions using large language models

Abstract

Detailed social determinants of health (SDoH) is often buried within clinical text in EHRs. Most current NLP efforts for SDoH have limitations, investigating limited factors, deriving data from a single institution, using specific patient cohorts/note types, with reduced focus on generalizability. We aim to address these issues by creating cross-institutional corpora and developing and evaluating the generalizability of classification models, including large language models (LLMs), for detecting SDoH factors using data from four institutions. Clinical notes were annotated with 21 SDoH factors at two levels: level 1 (SDoH factors only) and level 2 (SDoH factors and associated values). Compared to other models, instruction tuned LLM achieved top performance with micro-averaged F1 over 0.9 on level 1 corpora and over 0.84 on level 2 corpora. While models performed well when trained and tested on individual datasets, cross-dataset generalization highlighted remaining obstacles. Access to trained models will be made available at https://github.com/BIDS-Xu-Lab/LLMs4SDoH .

Fig. 1. Distribution of SDoH factors.

Number of sentences documenting each SDoH factor for all four corpora annotated at level 1 (identifying SDoH factors only).

Fig. 2. Heatmap of cross-dataset performance evaluation (level 1).

The diagonal shows micro-averaged F1scores when trained and tested on the same dataset for level 1 annotations. Other cells show F1 scores when trained on one dataset and tested on another dataset.

Fig. 3. Heatmap of cross-dataset performance evaluation (level 2).

The diagonal shows micro-averaged F1scores when trained and tested on the same dataset for level 2 annotations. Other cells show F1 scores when trained on one dataset and tested on another dataset.

Fig. 4. A schematic representation of the workflow.

Figure shows some of the SDoH factors, data sources, models used and evaluation process.