Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Apr;43(4):935-941.
doi: 10.1002/nau.25442. Epub 2024 Mar 7.

Conformity of ChatGPT recommendations with the AUA/SUFU guideline on postprostatectomy urinary incontinence

Vicktor B P Pinto  1 , Matheus F de Azevedo  1 , Marcelo L Wroclawski  2   3   4 , Guilherme Gentile  1 , Vinicius L M Jesus  1 , Jose de Bessa Junior  5 , William C Nahas  1 , Carlos A R Sacomani  6 , Jaspreet S Sandhu  7 , Cristiano M Gomes  1
Affiliations

Conformity of ChatGPT recommendations with the AUA/SUFU guideline on postprostatectomy urinary incontinence

Vicktor B P Pinto et al. Neurourol Urodyn. 2024 Apr.

Abstract

Introduction: Artificial intelligence (AI) shows immense potential in medicine and Chat generative pretrained transformer (ChatGPT) has been used for different purposes in the field. However, it may not match the complexity and nuance of certain medical scenarios. This study evaluates the accuracy of ChatGPT 3.5 and 4 in providing recommendations regarding the management of postprostatectomy urinary incontinence (PPUI), considering The Incontinence After Prostate Treatment: AUA/SUFU Guideline as the best practice benchmark.

Materials and methods: A set of questions based on the AUA/SUFU Guideline was prepared. Queries included 10 conceptual questions and 10 case-based questions. All questions were open and entered into the ChatGPT with a recommendation to limit the answer to 200 words, for greater objectivity. Responses were graded as correct (1 point); partially correct (0.5 point), or incorrect (0 point). Performances of versions 3.5 and 4 of ChatGPT were analyzed overall and separately for the conceptual and the case-based questions.

Results: ChatGPT 3.5 scored 11.5 out of 20 points (57.5% accuracy), while ChatGPT 4 scored 18 (90.0%; p = 0.031). In the conceptual questions, ChatGPT 3.5 provided accurate answers to six questions along with one partially correct response and three incorrect answers, with a final score of 6.5. In contrast, ChatGPT 4 provided correct answers to eight questions and partially correct answers to two questions, scoring 9.0. In the case-based questions, ChatGPT 3.5 scored 5.0, while ChatGPT 4 scored 9.0. The domains where ChatGPT performed worst were evaluation, treatment options, surgical complications, and special situations.

Conclusion: ChatGPT 4 demonstrated superior performance compared to ChatGPT 3.5 in providing recommendations for the management of PPUI, using the AUA/SUFU Guideline as a benchmark. Continuous monitoring is essential for evaluating the development and precision of AI-generated medical information.

Keywords: ChatGPT; artificial intelligence; prostatectomy; urinary incontinence; urology.

PubMed Disclaimer

References

REFERENCES

    1. Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29(8):1930‐1940. doi:10.1038/s41591-023-02448-8
    1. Borkowski AA. Applications of ChatGPT and large language models in medicine and health care: benefits and pitfalls. Fed Pract. 2023;40(6):170‐173. doi:10.12788/fp.0386
    1. Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intel. 2023;6:1169595. doi:10.3389/frai.2023.1169595
    1. Moons P, Van Bulck L. ChatGPT: can artificial intelligence language models be of value for cardiovascular nurses and allied health professionals. Eur J Cardiovasc Nurs. 2023;22:e55‐e59. doi:10.1093/eurjcn/zvad022
    1. Kung JE, Marshall C, Gauthier C, Gonzalez TA, Jackson JB. Evaluating ChatGPT performance on the orthopaedic in‐training examination. JBJS Open Access. 2023;8(3):e23.0005. doi:10.2106/jbjs.oa.23.00056

LinkOut - more resources