ChatGPT as a Clinical Decision Maker for Urolithiasis: Compliance with the Current European Association of Urology Guidelines

Abstract

Background and objective: Generative artificial intelligence models are among the most promising and widely used tools used in health care. This review investigates GPT-4 answers to decision-making questions regarding the diagnosis and treatment of urolithiasis across several clinical settings and their correspondence to the current European Association of Urology (EAU) guidelines.

Methods: In March 2024, the GPT-4 model was asked 11 questions, containing a brief description of a patient with urolithiasis. All questions were grouped according to urolithiasis care step: diagnosis, urgent care, scheduled intervention, and metaphylaxis. When responses were received, compliance with the current EAU guidelines was assessed by experienced urologists.

Key findings and limitations: Although all responses were provided with information that corresponded to EAU guidelines, six of the 11 answers were associated with missed guideline-provided parts, and incorrect data were given in eight of the 11 answers. GPT-4 is relatively safe in the initial diagnostic flow of patients suspected of having stones within the urinary tract and during treatment planning; however, its understanding of all the nuances of metaphylaxis leaves much to be desired and is far from the dogma given in the EAU guidelines. Moreover, GPT-4 knowledge of strategy and algorithm is not always aligned with the EAU guidelines.

Conclusions and clinical implications: Despite the fact that from the perspective of patients with urolithiasis, GPT-4 is capable of answering their questions well, the specificity of questions from urologists is labor intensive for its current version, and necessitates the ability to interpret it correctly and further attempts to improve it. While some aspects of diagnostics are more accurate, these struggle with surgical planning and algorithms in line with the EAU guidelines.

Patient summary: The generative artificial intelligence (AI) model GPT-4 is capable of answering urology-related questions, but lacks detailed responses. Although some aspects of the diagnostics are accurate, knowledge of surgical planning is not in line with the European Association of Urology guidelines. Future improvements should focus on efforts to enhance the accuracy, reliability, and clinical relevance of AI tools in urology.

Keywords: Clinical decision; Diagnosis; Generative pretrained transformer; Treatment; Urolithiasis.