Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
[Preprint]. 2023 Feb 28:rs.3.rs-2566942.
doi: 10.21203/rs.3.rs-2566942/v1.

Assessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT Model

Douglas Johnson  1 , Rachel Goodman  2 , J Patrinely  1 , Cosby Stone  3 , Eli Zimmerman  1 , Rebecca Donald  1 , Sam Chang  1 , Sean Berkowitz  1 , Avni Finn  1 , Eiman Jahangir  1 , Elizabeth Scoville  1 , Tyler Reese  1 , Debra Friedman  1 , Julie Bastarache  1 , Yuri van der Heijden  1 , Jordan Wright  1 , Nicholas Carter  1 , Matthew Alexander  1 , Jennifer Choe  1 , Cody Chastain  1 , John Zic  1 , Sara Horst  1 , Isik Turker  1 , Rajiv Agarwal  1 , Evan Osmundson  1 , Kamran Idrees  1 , Colleen Kieman  1 , Chandrasekhar Padmanabhan  1 , Christina Bailey  1 , Cameron Schlegel  1 , Lola Chambless  4 , Mike Gibson  1 , Travis Osterman  1 , Lee Wheless  1
Affiliations

Assessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT Model

Douglas Johnson et al. Res Sq. .

Abstract

Background: Natural language processing models such as ChatGPT can generate text-based content and are poised to become a major information source in medicine and beyond. The accuracy and completeness of ChatGPT for medical queries is not known.

Methods: Thirty-three physicians across 17 specialties generated 284 medical questions that they subjectively classified as easy, medium, or hard with either binary (yes/no) or descriptive answers. The physicians then graded ChatGPT-generated answers to these questions for accuracy (6-point Likert scale; range 1 - completely incorrect to 6 - completely correct) and completeness (3-point Likert scale; range 1 - incomplete to 3 - complete plus additional context). Scores were summarized with descriptive statistics and compared using Mann-Whitney U or Kruskal-Wallis testing.

Results: Across all questions (n=284), median accuracy score was 5.5 (between almost completely and completely correct) with mean score of 4.8 (between mostly and almost completely correct). Median completeness score was 3 (complete and comprehensive) with mean score of 2.5. For questions rated easy, medium, and hard, median accuracy scores were 6, 5.5, and 5 (mean 5.0, 4.7, and 4.6; p=0.05). Accuracy scores for binary and descriptive questions were similar (median 6 vs. 5; mean 4.9 vs. 4.7; p=0.07). Of 36 questions with scores of 1-2, 34 were re-queried/re-graded 8-17 days later with substantial improvement (median 2 vs. 4; p<0.01).

Conclusions: ChatGPT generated largely accurate information to diverse medical queries as judged by academic physician specialists although with important limitations. Further research and model development are needed to correct inaccuracies and for validation.

Keywords: ChatGPT; artificial intelligence; clinical decision making; deep learning; knowledge dissemination; large language model; medical education; natural language processing.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Methodology *DBJ and LEW scored two separate datasets of melanoma and immunotherapy and common conditions questions.
Figure 2
Figure 2
Accuracy of Chat-GPT-Generated Answers Accuracy of AI answers from multispecialty questions (A-C) or all questions (multispecialty, melanoma and immunotherapy, and common medical conditions; D-F). *p < 0.01, ** p = 0.03
See this image and copyright information in PMC

References

    1. Shen Y, Heacock L, Elias J, Hentel KD, Reig B, Shih G, et al. ChatGPT and Other Large Language Models Are Double-edged Swords. Radiological Society of North America; 2023. p. 230163. - PubMed
    1. Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal R et al. Language models are few-shot learners. Advances in neural information processing systems. 2020;33:1877–901.
    1. Christiano PF, Leike J, Brown T, Martic M, Legg S, Amodei D. Deep reinforcement learning from human preferences. Advances in neural information processing systems. 2017;30.
    1. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models. medRxiv. 2022:2022.12.19.22283643. - PMC - PubMed
    1. Gilson A, Safranek C, Huang T, Socrates V, Chi L, Taylor RA, et al. How Well Does ChatGPT Do When Taking the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. medRxiv. 2022:2022.12. 23.22283901. - PubMed

Publication types

LinkOut - more resources