Dedicated AI Expert System vs Generative AI With Large Language Model for Clinical Diagnoses

Abstract

Importance: Large language models (LLMs) have not yet been compared with traditional diagnostic decision support systems (DDSSs) on unpublished clinical cases.

Objective: To compare the performance of 2 widely used LLMs (ChatGPT, version 4 [hereafter, LLM1] and Gemini, version 1.5 [hereafter, LLM2]) with a DDSS (DXplain [hereafter, DDSS]) on 36 unpublished general medicine cases.

Design, setting, and participants: This diagnostic study, conducted from October 6, 2023, to November 22, 2024, looked for the presence of the known case diagnosis in the differential diagnoses of the LLMs and DDSS after data from previously unpublished clinical cases from 3 academic medical centers were entered. The systems' performance was assessed both with and without laboratory test data. Each case was reviewed by 3 physicians blinded to the case diagnosis. Physicians identified all clinical findings as well as the subset deemed relevant to making the diagnosis for mapping to the DDSS's controlled vocabulary. Two other physicians, also blinded to the diagnoses, entered the data from these cases into the DDSS, LLM1, and LLM2.

Exposures: All cases were entered into each LLM twice, with and without laboratory test results. For the DDSS, each case was entered 4 times: for all findings and for findings relevant to the diagnosis, each with and without laboratory test results. The top 25 diagnoses in each resulting differential diagnosis were reviewed.

Main outcomes and measures: Presence or absence of the case diagnosis in the system's differential diagnosis and, when present, in which quintile it appeared in the top 25 diagnoses.

Results: Among 36 patient cases of various races and ethnicities, genders, and ages (mean [SD] age, 51.4 [16.4] years), in the version with all findings but no laboratory test results, the DDSS listed the case diagnosis in its differential diagnosis more often (56% [20 of 36]) than LLM1 (42% [15 of 36]) and LLM2 (39% [14 of 36]), although this difference did not reach statistical significance (DDSS vs LLMI, P = .09; DDSS vs LLM2, P = .08). All 3 systems listed the case diagnosis in most cases if laboratory test results were included (all findings DDSS, 72% [26 of 36]; LLM1, 64% [23 of 36]; and LLM2, 58% [21 of 36]).

Conclusions and relevance: In this diagnostic study comparing the performance of a traditional DDSS and current LLMs on unpublished clinical cases, in most cases, every system listed the case diagnosis in their top 25 diagnoses if laboratory test results were included. A hybrid approach that combines the parsing and expository linguistic capabilities of LLMs with the deterministic and explanatory capabilities of traditional DDSSs may produce synergistic benefits.

Figure.. Diagnostic Decision Support System (DDSS) vs Large Language Models (LLMs)

Comparison of performance across systems for placing the correct diagnosis higher up on a differential diagnosis consisting of 25 diagnoses. DDSS ALL indicates the DDSS with all clinical data; DDSS REL, the DDSS with only clinical data considered relevant.