Influence of medical educational background on the diagnostic quality of ChatGPT-4 responses in internal medicine: A pilot study

Abstract

This pilot study evaluated the influence of medical background on the diagnostic quality of ChatGPT-4's responses in Internal Medicine. Third-year students, residents and specialists summarised five complex NEJM clinical cases before querying ChatGPT-4. Diagnostic ranking, assessed by independent experts, revealed that residents significantly outperformed students (OR 2.33, p = .007); though overall performance was low. These findings indicate that user expertise and concise case summaries are critical for optimising AI diagnostics, highlighting the need for enhanced AI training and user interaction strategies.

Keywords: ChatGPT‐4; artificial intelligence; clinical decision making; diagnostic ranking; internal medicine; large language models.

FIGURE 1

Study design flowchart.

FIGURE 2

Interrater agreement and study results. (A) Scatter plots of two‐by‐two rater scores for the variable ‘diagnostic ranking’; the red dashed line represents the least square regression, with grey shades highlighting the 95% confidence intervals of the correlations. Spearman's rank coefficients are .91, .91 and .92, respectively. (B) Violin plots of the diagnostic ranking scores according to the evaluating groups; grey dots represent individual diagnostic ranking scores assigned by the independent raters, whereas the thick red dots represent group means. Data distributions are highlighted by the steel blue shades. (C) Forest plot of odds ratios for each represented contrast of groups; *p‐value <.05. Residents versus Students: OR 2.33, 95% confidence interval (CI) = 1.27–4.28, p‐value = .007. Consultants versus Students: OR 1.42, 95% CI = .77–2.61, p‐value = .258.