Assessing LLMs on IDSA Practice Guidelines for the Diagnosis and Treatment of Native Vertebral Osteomyelitis: A Comparison Study

Abstract

Background: Native vertebral osteomyelitis (NVO) presents diagnostic and therapeutic challenges requiring adherence to complex clinical guidelines. The emergence of large language models (LLMs) offers new avenues for real-time clinical decision support, yet their utility in managing NVO has not been formally assessed. Methods: This study evaluated four LLMs-Consensus, Gemini, ChatGPT-4o Mini, and ChatGPT-4o-using 13 standardized questions derived from the 2015 IDSA guidelines. Each model generated 13 responses (n = 52), which were independently assessed by three orthopedic surgeons for accuracy (4-point scale) and comprehensiveness (five-point scale). Results: ChatGPT-4o produced the longest responses (428.0 ± 45.4 words), followed by ChatGPT-4o Mini (392.2 ± 97.4), Gemini (358.2 ± 60.5), and Consensus (213.2 ± 68.8). Accuracy ratings showed that ChatGPT-4o and Gemini achieved the highest proportion of "Excellent" responses (54% and 51%, respectively), while Consensus received only 20%. Comprehensiveness scores mirrored this trend, with ChatGPT-4o (3.95 ± 0.79) and Gemini (3.82 ± 0.68) significantly outperforming Consensus (2.87 ± 0.66). Domain-specific analysis revealed that ChatGPT-4o achieved a 100% "Excellent" accuracy rating in therapy-related questions. Statistical analysis confirmed significant inter-model differences (p < 0.001). Conclusions: Advanced LLMs-especially ChatGPT-4o and Gemini-demonstrated high accuracy and depth in interpreting clinical guidelines for NVO. These findings highlight their potential as effective tools in augmenting evidence-based decision-making and improving consistency in clinical care.

Keywords: ChatGPT; Gemini; IDSA guidelines; artificial intelligence; clinical decision support; large language models; native vertebral osteomyelitis; spine infection.

Figure 1

Flowchart of the overall study design.

Figure 2

Average accuracy scores of LLM-chatbot replies to ISDA-related queries, as determined by the three orthopaedic graders. Heatmap (A) of individual accuracy score, (B) average scores for accuracy, and (C) average comprehensiveness scores. The asterisks indicating statistical significance.

Figure 3

Consensus-based accuracy ratings of LLM-chatbot responses to IDSA practice guidelines related questions. LLM = Large language model. IDSA = Infectious Diseases Society of America.