Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks

Abstract

It is likely that individuals are turning to Large Language Models (LLMs) to seek health advice, much like searching for diagnoses on Google. We evaluate clinical accuracy of GPT-3·5 and GPT-4 for suggesting initial diagnosis, examination steps and treatment of 110 medical cases across diverse clinical disciplines. Moreover, two model configurations of the Llama 2 open source LLMs are assessed in a sub-study. For benchmarking the diagnostic task, we conduct a naïve Google search for comparison. Overall, GPT-4 performed best with superior performances over GPT-3·5 considering diagnosis and examination and superior performance over Google for diagnosis. Except for treatment, better performance on frequent vs rare diseases is evident for all three approaches. The sub-study indicates slightly lower performances for Llama models. In conclusion, the commercial LLMs show growing potential for medical question answering in two successive major releases. However, some weaknesses underscore the need for robust and regulated AI models in health care. Open source LLMs can be a viable option to address specific needs regarding data privacy and transparency of training.

Fig. 1. Performance comparison of GPT-3·5 vs GPT-4 vs Google.

a Performance of GPT-3·5 vs GPT-4 vs Google for diagnosis. b Performance of GPT-3·5 vs GPT-4 for examination (exact adjusted p-value p = 3.2241·10⁻⁶). c Performance of GPT-3·5 vs GPT-4 for treatment. Bubble plots show the pairwise comparison of two approaches. Cumulative frequency plots show the cumulative number of cases (Y-axis) and their accuracy scores (X-axis) for each disease frequency subgroup (light blue: rare, intermediate blue: less frequent, dark blue: frequent). One-sided Mann-Whitney test was applied for statistical testing (adjusted with Bonferroni correction for multiple testing considering n = 12 tests for diagnosis, n = 7 tests for examination and treatment).

Fig. 2. Performance comparison of GPT-3·5 vs GPT-4 vs Ll2-7B vs Ll2-70B considering top-3 and bottom-3 cases.

Black dots mark the top-3 cases based on GPT-4’s cumulative score for rare, less frequent and frequent diseases. Red dots the bottom-3 cases. Violin plots visualize the performance of GPT-3·5 and GPT-4 for all n = 110 cases. Ll2-7B: Llama-2-7b-chat; LL2-70B: Llama-2-70b-chat.

Fig. 3. Overview of process steps.

Cases from clinical case books were filtered and processed to generate patient queries for GPT-3·5 and GPT-4. The answers on suspected diagnosis, examination and treatment options were evaluated by two independent physicians and rated on a 5-point Likert scale.