Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Aug 23;15(8):e43958.
doi: 10.7759/cureus.43958. eCollection 2023 Aug.

Artificial Intelligence (AI) in Radiology: A Deep Dive Into ChatGPT 4.0's Accuracy with the American Journal of Neuroradiology's (AJNR) "Case of the Month"

Pokhraj P Suthar  1 , Avin Kounsal  1 , Lavanya Chhetri  2 , Divya Saini  3 , Sumeet G Dua  1
Affiliations

Artificial Intelligence (AI) in Radiology: A Deep Dive Into ChatGPT 4.0's Accuracy with the American Journal of Neuroradiology's (AJNR) "Case of the Month"

Pokhraj P Suthar et al. Cureus. .

Abstract

The advent of artificial intelligence (AI), particularly large language models (LLMs) such as ChatGPT 4.0, holds significant potential in healthcare, specifically in radiology. This study examined the accuracy of ChatGPT 4.0 (July 20, 2023, version) in solving diagnostic quizzes from the American Journal of Neuroradiology's (AJNR) "Case of the Month." We evaluated the diagnostic accuracy of ChatGPT 4.0 when provided with a patient's history and imaging findings weekly over four weeks, using 140 cases from the AJNR "Case of the Month" portal (from November 2011 to July 2023). The overall diagnostic accuracy was found to be 57.86% (81 out of 140 cases). The diagnostic performance varied across brain, head and neck, and spine subgroups, with accuracy rates of 54.65%, 67.65%, and 55.0%, respectively. These findings suggest that AI models such as ChatGPT 4.0 could serve as useful adjuncts in radiological diagnostics, thus potentially enhancing patient care and revolutionizing medical education.

Keywords: accuracy; artificial intelligence in radiology; chat gpt; large language models (llms); neuroradiology.

PubMed Disclaimer

Conflict of interest statement

The authors have declared that no competing interests exist.

Figures

Figure 1
Figure 1. Study overview. This study evaluates the accuracy of the GPT-4 based ChatGPT (July 20, 2023, version) in solving diagnostic quizzes presented in the American Journal of Neuroradiology's (AJNR) "Case of the Month."
Figure 2
Figure 2. Study outcome overview. This study demonstrates how the diagnostic accuracy of ChatGPT increases incrementally with the stepwise addition of historical data and weekly findings. The overall diagnostic accuracy of ChatGPT for resolving the AJNR's "Case of the Month" is 57.86% (81 out of 140 cases).
Figure 3
Figure 3. Study outcome overview. This study demonstrates how the diagnostic accuracy of ChatGPT varied across the subgroups. For the brain category, ChatGPT demonstrated an accuracy of 54.65% (47 out of 86 cases). In the head and neck category, the accuracy climbed to 67.65% (23 out of 34 cases), while in the spine category, the system delivered an accuracy of 55.0% (11 out of 20 cases).
See this image and copyright information in PMC

References

    1. Advances in natural language processing. Hirschberg J, Manning CD. Science. 2015;349:261–266. - PubMed
    1. News verifiers showdown: a comparative performance evaluation of ChatGPT 3.5, ChatGPT 4.0, Bing AI, and Bard in news fact-checking. arXic [cs.CL. [ Aug; 2023 ]. 2023. https://fliphtml5.com/gnel/izuf/News_Verifiers_Showdown%3A_A_Comparative... https://fliphtml5.com/gnel/izuf/News_Verifiers_Showdown%3A_A_Comparative...
    1. GPT-4 technical report. arXiv [cs.CL. [ Aug; 2023 ]. 2023. https://cdn.openai.com/papers/gpt-4.pdf https://cdn.openai.com/papers/gpt-4.pdf
    1. ChatGPT’s diagnostic performance from patient history and imaging findings on the diagnosis please quizzes. Ueda D, Mitsuyama Y, Takita H, Horiuchi D, Walston SL, Tatekawa H, Miki Y. Radiology. 2023;308:0. - PubMed
    1. ChatGPT's quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. Hoch CC, Wollenberg B, Lüers JC, et al. Eur Arch Otorhinolaryngol. 2023;280:4271–4278. - PMC - PubMed

LinkOut - more resources