Can Patients With Urogenital Cancer Rely on Artificial Intelligence Chatbots for Treatment Decisions?

Abstract

Objectives: In the era of artificial intelligence, almost half of the patients use the internet to get information about their diseases. Our study aims to demonstrate the reliability of the information provided by artificial intelligence chatbots (AICs) about urogenital cancer treatments.

Methods: The most frequently searched keyword about prostate, bladder, kidney, and testicular cancer treatment via Google Trends was asked to 3 different AICs (ChatGPT, Gemini, Copilot). The answers were evaluated by 5 different examiners in terms of readability, understandability, actionability, reliability, and transparency.

Results: The DISCERN score evaluation indicates that ChatGPT and Gemini provided moderate quality information, while Copilot's quality was low. (Total DISCERN scores; 41, 42, 35, respectively). PEMAT-P Understandability scores were low (40%) and PEMAT-P Actionability scores were moderate only for Gemini (60%) and low for the others (40%). Their readability according to the Coleman-Liau index was above the college level (16.9, 17.2, 16, respectively).

Conclusions: In the era of artificial intelligence, patients will inevitably use AICs due to their easy and fast accessibility. However, patients need to recognize that AICs do not provide stage-specific treatment options, but only moderate-quality, low-reliability information about the disease, as well as information that is very difficult to read.

Keywords: Artificial Intelligence; ChatGPT; Quality assessment; Treatment; Urogenital neoplasms.