Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Mar;27(1):103-108.
doi: 10.1038/s41391-023-00705-y. Epub 2023 Jul 29.

Quality of information and appropriateness of ChatGPT outputs for urology patients

Andrea Cocci  1 , Marta Pezzoli  2 , Mattia Lo Re  2 , Giorgio Ivan Russo  3 , Maria Giovanna Asmundo  3 , Mikkel Fode  4   5 , Giovanni Cacciamani  6 , Sebastiano Cimino  3 , Andrea Minervini  2 , Emil Durukan  5
Affiliations

Quality of information and appropriateness of ChatGPT outputs for urology patients

Andrea Cocci et al. Prostate Cancer Prostatic Dis. 2024 Mar.

Abstract

Background: The proportion of health-related searches on the internet is continuously growing. ChatGPT, a natural language processing (NLP) tool created by OpenAI, has been gaining increasing user attention and can potentially be used as a source for obtaining information related to health concerns. This study aims to analyze the quality and appropriateness of ChatGPT's responses to Urology case studies compared to those of a urologist.

Methods: Data from 100 patient case studies, comprising patient demographics, medical history, and urologic complaints, were sequentially inputted into ChatGPT, one by one. A question was posed to determine the most likely diagnosis, suggested examinations, and treatment options. The responses generated by ChatGPT were then compared to those provided by a board-certified urologist who was blinded to ChatGPT's responses and graded on a 5-point Likert scale based on accuracy, comprehensiveness, and clarity as criterias for appropriateness. The quality of information was graded based on the section 2 of the DISCERN tool and readability assessments were performed using the Flesch Reading Ease (FRE) and Flesch-Kincaid Reading Grade Level (FKGL) formulas.

Results: 52% of all responses were deemed appropriate. ChatGPT provided more appropriate responses for non-oncology conditions (58.5%) compared to oncology (52.6%) and emergency urology cases (11.1%) (p = 0.03). The median score of the DISCERN tool was 15 (IQR = 5.3) corresponding to a quality score of poor. The ChatGPT responses demonstrated a college graduate reading level, as indicated by the median FRE score of 18 (IQR = 21) and the median FKGL score of 15.8 (IQR = 3).

Conclusions: ChatGPT serves as an interactive tool for providing medical information online, offering the possibility of enhancing health outcomes and patient satisfaction. Nevertheless, the insufficient appropriateness and poor quality of the responses on Urology cases emphasizes the importance of thorough evaluation and use of NLP-generated outputs when addressing health-related concerns.

PubMed Disclaimer

References

    1. Wise J. How Many People Use the Internet Daily in 2023? - EarthWeb. https://earthweb.com/how-many-people-use-the-internet-daily/ (accessed 15 May2023).
    1. NTIA. More than Half of American Households Used the Internet for Health-Related Activities in 2019, NTIA Data Show | National Telecommunications and Information Administration. https://www.ntia.gov/blog/2020/more-half-american-households-used-intern... (accessed 2 May2023).
    1. Eysenbach G, Kohler C. What is the prevalence of health-related searches on the World Wide Web? Qualitative and quantitative analysis of search engine queries on the Internet. AMIA Annu Symp Proc. 2003;2003:225. - PubMed - PMC
    1. Introducing ChatGPT. https://openai.com/blog/chatgpt (accessed 15 May2023).
    1. Liu Y, Yang Z, Yu Z, Liu Z, Liu D, Lin H, et al. Generative artificial intelligence and its applications in materials science: Current situation and future perspectives. J Mater. 2023. https://doi.org/10.1016/J.JMAT.2023.05.001 .

LinkOut - more resources