Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Mar 1;7(2):pkad015.
doi: 10.1093/jncics/pkad015.

Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information

Skyler B Johnson  1   2 , Andy J King  2   3 , Echo L Warner  2   4 , Sanjay Aneja  5   6 , Benjamin H Kann  7 , Carma L Bylund  8
Affiliations

Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information

Skyler B Johnson et al. JNCI Cancer Spectr. .

Abstract

Data about the quality of cancer information that chatbots and other artificial intelligence systems provide are limited. Here, we evaluate the accuracy of cancer information on ChatGPT compared with the National Cancer Institute's (NCI's) answers by using the questions on the "Common Cancer Myths and Misconceptions" web page. The NCI's answers and ChatGPT answers to each question were blinded, and then evaluated for accuracy (accurate: yes vs no). Ratings were evaluated independently for each question, and then compared between the blinded NCI and ChatGPT answers. Additionally, word count and Flesch-Kincaid readability grade level for each individual response were evaluated. Following expert review, the percentage of overall agreement for accuracy was 100% for NCI answers and 96.9% for ChatGPT outputs for questions 1 through 13 (ĸ = ‒0.03, standard error = 0.08). There were few noticeable differences in the number of words or the readability of the answers from NCI or ChatGPT. Overall, the results suggest that ChatGPT provides accurate information about common cancer myths and misconceptions.

PubMed Disclaimer

Conflict of interest statement

All authors had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Skyler B. Johnson, MD, has no financial conflicts of interest. All other researchers reported no financial conflicts of interests.

References

    1. Calixte R, Rivera A, Oridota O, Beauchamp W, Camacho-Rivera M.. Social and demographic patterns of health-related Internet use among adults in the United States: a secondary data analysis of the Health Information National Trends Survey. Int J Environ Res Public Health. 2020;17(18):6856. - PMC - PubMed
    1. Johnson SB, Parsons M, Dorff T, et al.Cancer misinformation and harmful information on Facebook and other social media: a brief report. J Natl Cancer Inst. 2022;114(7):1036–1039. - PMC - PubMed
    1. Miner AS, Laranjo L, Kocaballi AB.. Chatbots in the fight against the COVID-19 pandemic. NPJ Digit Med. 2020;3:65. - PMC - PubMed
    1. Common cancer myths and misconceptions. National Cancer Institute. Updated December 6, 2022. Accessed December 16, 2022. https://www.cancer.gov/about-cancer/causes-prevention/risk/myths
    1. Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bulletin. 1971;76(5):378-382.

Publication types

LinkOut - more resources