Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Sep 18:15589447251371089.
doi: 10.1177/15589447251371089. Online ahead of print.

Do ChatGPT and Gemini's Recommendations Align With Established Guidelines for Hand and Upper Extremity Surgery?

Yibin B Zhang  1 , Fielding S Fischer  1 , Matthew V Abola  2 , Daniel A Osei  2 , Scott W Wolfe  2 , Troy B Amen  2
Affiliations

Do ChatGPT and Gemini's Recommendations Align With Established Guidelines for Hand and Upper Extremity Surgery?

Yibin B Zhang et al. Hand (N Y). .

Abstract

Background: The use of large language models (LLMs) such as ChatGPT and Gemini in clinical settings has surged, presenting potential benefits in reducing administrative workload and enhancing patient communication. However, concerns about the clinical accuracy of these tools persist. This study evaluated the concordance of ChatGPT and Gemini's recommendations with American Academy of Orthopedic Surgeons (AAOS) clinical practice guidelines (CPGs) for carpal tunnel syndrome, distal radius fractures, and glenohumeral joint osteoarthritis.

Methods: ChatGPT (version 4o) and Gemini (version 1.5 Flash) were queried using structured text-based prompts aligned with AAOS CPGs. The LLMs' outputs were analyzed by blinded reviewers to determine concordance with the guidelines. Concordance rates were compared across models, topics, and guideline strength using descriptive statistics and McNemar's test. The transparency of responses, including source citation, was also assessed.

Results: A total of 174 recommendations were generated, with an overall concordance rate of 62.1%. When comparing concordance rates between LLMs, there was no statistically significant difference between ChatGPT and Gemini (66.7% vs 57.5%, P = .131). Concordance varied by topic and guideline strength, with ChatGPT performing best for moderately supported guidelines. Both models demonstrated low citation transparency. Gemini provided sources for 39.1% of recommendations, significantly more than ChatGPT's 3.5% (P < .0001).

Conclusions: Despite modest concordance rates, both models exhibited significant limitations, including variability across topics and guideline strengths, as well as insufficient citation transparency. These findings highlight the challenges in integrating LLMs into clinical practice and emphasize the need for further refinement and evaluation before adoption in hand surgery.

Keywords: CPGs (clinical practice guidelines); LLMs (large language models); artificial intelligence in healthcare; clinical accuracy and transparency; clinical decision-making.

PubMed Disclaimer

Conflict of interest statement

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Figures

hand and upper extremity surgery treatment topic. bar chart comparing concordance of ChatGPT and Gemini recommendations to AAOS CPGs stratified by topic.The data includes the concordance percentages for ChatGPT and Gemini regarding the treatment for Carpal Tunnel Syndrome, Shoulder OA, and Distal Radius Fracture.
Figure 1.
Hand and upper extremity surgery topic. Bar graph comparing concordance of ChatGPT and Gemini recommendations to AAOS CPGs stratified by topic. Note. AAOS = American Academy of Orthopedic Surgeons, CPG = clinical practice guidelines, OA = osteoarthritis.
Create a bar graph illustrating the concordance rate between ChatGPT and Gemini’s suggestions on AAOS guidelines divided into Strong, Moderate, Limited and Consensus categories.
Figure 2.
Guideline strength. Bar graph comparing concordance of ChatGPT and Gemini recommendations to AAOS CPGs stratified by strength of the guideline. Note. AAOS = American Academy of Orthopedic Surgeons, CPG = clinical practice guidelines.
See this image and copyright information in PMC

References

    1. Browne R, Gull K, Hurley CM, Sugrue RM, O’Sullivan JB. ChatGPT-4 can help hand surgeons communicate better with patients. J Hand Surg Glob Online. 2024;6(3):436-438. - PMC - PubMed
    1. Yao JJ, Aggarwal M, Lopez RD, Namdari S. Current concepts review: large language models in orthopaedics: definitions, uses, and limitations. J Bone Joint Surg Am. 2024;106(15):1411-1418. - PubMed
    1. Pressman SM, Borna S, Gomez-Cabello CA, Haider SA, Forte AJ. AI in hand surgery: assessing large language models in the classification and management of hand injuries. J Clin Med. 2024;13(10):2832. - PMC - PubMed
    1. Liu S, McCoy AB, Wright AP, et al. Leveraging large language models for generating responses to patient messages-a subjective analysis. J Am Med Inform Assoc. 2024;31(6):1367-1379. - PMC - PubMed
    1. Singh S, Djalilian A, Ali MJ. ChatGPT and ophthalmology: exploring its potential with discharge summaries and operative notes. Semin Ophthalmol. 2023;38(5):503-507. - PubMed

LinkOut - more resources