Assessing the Accuracy of Diagnostic Capabilities of Large Language Models

Abstract

Background: In recent years, numerous artificial intelligence applications, especially generative large language models, have evolved in the medical field. This study conducted a structured comparative analysis of four leading generative large language models (LLMs)-ChatGPT-4o (OpenAI), Grok-3 (xAI), Gemini-2.0 Flash (Google), and DeepSeek-V3 (DeepSeek)-to evaluate their diagnostic performance in clinical case scenarios. Methods: We assessed medical knowledge recall and clinical reasoning capabilities through staged, progressively complex cases, with responses graded by expert raters using a 0-5 scale. Results: All models performed better on knowledge-based questions than on reasoning tasks, highlighting the ongoing limitations in contextual diagnostic synthesis. Overall, DeepSeek outperformed the other models, achieving significantly higher scores across all evaluation dimensions (p < 0.05), particularly in regards to medical reasoning tasks. Conclusions: While these findings support the feasibility of using LLMs for medical training and decision support, the study emphasizes the need for improved interpretability, prompt optimization, and rigorous benchmarking to ensure clinical reliability. This structured, comparative approach contributes to ongoing efforts to establish standardized evaluation frameworks for integrating LLMs into diagnostic workflows.

Keywords: artificial intelligence; diagnostic accuracy; large language modes; medical education.

Figure 1

Flowchart of the methodological steps used in the study.

Figure 2

Conceptual clinical case stage/context complexity and structure.

Figure 3

Question type distribution for each clinical case.

Figure 4

LLMs comparison of mean performance scores (±SD) according to the five criteria used to assess diagnostic capabilities (CG = Chat GPT, GK = Grok, GE = Gemini, and DS = DeepSeek).

Figure 5

LLMs comparison of mean performance scores (±SD) for questions regarding medical knowledge, according to the five criteria (CG = Chat GPT, GK = Grok, GE = Gemini, and DS = DeepSeek).

Figure 6

LLMs comparison of mean performance scores (±SD) for questions regarding medical reasoning, according to the five criteria (CG = Chat GPT, GK = Grok, GE = Gemini, and DS = DeepSeek).