Large language models outperform traditional structured data-based approaches in identifying immunosuppressed patients

Abstract

Identifying immunosuppressed patients using structured data can be challenging. Large language models effectively extract structured concepts from unstructured clinical text. Here we show that GPT-4o outperforms traditional approaches in identifying immunosuppressive conditions and medication use by processing hospital admission notes. We also demonstrate the extensibility of our approach in an external dataset. Cost-effective models like GPT-4o mini and Llama 3.1 also perform well, but not as well as GPT-4o.

Fig. 1:. Flow diagram of study design comparing structured data and LLM performance in identifying immunosuppression followed by external validation of LLM performance.

Structured data and admission notes were extracted for 827 SCRIPT admissions which were adjudicated for immunosuppressive conditions and medications. The performance of structured data algorithms and LLMs in predicting the adjudicated labels was assessed. External validation of using LLMs to predict adjudicated immunosuppression labels was performed in 200 MIMIC-III admission notes.

Fig. 2:. Performance of structured data in identifying immunosuppressive conditions and medications.

F1 scores, precision, and recall were computed by comparing structured data predictions to adjudicated labels. A Performance metrics across an increasing minimum number of dates with a diagnosis code required to count as a case. B Confusion matrices for predicting medication use by the presence of a valid medication order in the six months prior to admission.

Fig. 3:. Prompts used for immunosuppression attribute extraction and estimation from admission notes.

Prompts for extracting A immunosuppressive conditions and B immunosuppressive medications where the highlighted portion is either a list of A conditions or B medications.

Fig. 4:. Performance of GPT-4o in identifying immunosuppressive conditions and medications in the SCRIPT cohort.

F1 scores, precision, and recall were computed by comparing GPT-4o predictions to adjudicated labels. Confusion matrices for GPT-4o predicting A immunosuppressive conditions and B immunosuppressive medication use from admission notes.

Fig. 5:. Performance of GPT-4o in identifying immunosuppressive conditions and medications in the MIMIC-III cohort.

F1 scores, precision, and recall were computed by comparing GPT-4o predictions to adjudicated labels. Confusion matrices for GPT-4o predicting A immunosuppressive conditions and B immunosuppressive medication use from admission notes.