Leveraging ChatGPT for Enhancing Learning in Radiology Resident Education

Abstract

Rationale and objectives: Chat generative pre-trained transformer (ChatGPT) is a generative artificial intelligence chatbot based on a LLM at the forefront of technological development with promising applications in medical education. This study aims to evaluate the use of ChatGPT in generating board-style practice questions for radiology resident education.

Materials and methods: Multiple-choice questions (MCQs) were generated by ChatGPT from resident lecture transcripts using a custom prompt. 17 of the ChatGPT-generated MCQs were selected for inclusion in the study and randomly combined with 11 attending radiologist-written MCQs. For each MCQ, the 21 participating radiology residents answered the MCQ, rated the MCQ from 1-10 on effectiveness in reinforcing lecture material, and responded whether they thought an attending radiologist at their institution wrote the MCQ versus an alternative source.

Results: Perceived MCQ quality was not significantly different between ChatGPT-generated (M=6.93, SD=0.29) and attending radiologist-written MCQs (M=7.08, SD=0.51) (p=0.15). MCQ correct answer percentages did not significantly differ between ChatGPT-generated (M=57%, SD=20%) and attending radiologist-written MCQs (M=59%, SD=25%) (p=0.78). The percentage of MCQs thought to be written by an attending radiologist was significantly different between ChatGPT-generated (M=57%, SD=13%) and attending radiologist-written MCQs (M=71%, SD=20%) (p=0.04).

Conclusion: LLMs such as ChatGPT demonstrate potential in generating and presenting educational material for radiology education, and their use should be explored further on a larger scale.

Keywords: Artificial intelligence; ChatGPT; Education; Multiple-choice question; Resident.