Assessing the Performance of Chatbots on the Taiwan Psychiatry Licensing Examination Using the Rasch Model

Abstract

Background/objectives: The potential and limitations of chatbots in medical education and clinical decision support, particularly in specialized fields like psychiatry, remain unknown. By using the Rasch model, our study aimed to evaluate the performance of various state-of-the-art chatbots on psychiatry licensing exam questions to explore their strengths and weaknesses.

Methods: We assessed the performance of 22 leading chatbots, selected based on LMArena benchmark rankings, using 100 multiple-choice questions from the 2024 Taiwan psychiatry licensing examination, a nationally standardized test required for psychiatric licensure in Taiwan. Chatbot responses were scored for correctness, and we used the Rasch model to evaluate chatbot ability.

Results: Chatbots released after February 2024 passed the exam, with ChatGPT-o1-preview achieving the highest score of 85. ChatGPT-o1-preview showed a statistically significant superiority in ability (p < 0.001), with a 1.92 logits improvement compared to the passing threshold. It demonstrated strengths in complex psychiatric problems and ethical understanding, yet it presented limitations in up-to-date legal updates and specialized psychiatry knowledge, such as recent amendments to the Mental Health Act, psychopharmacology, and advanced neuroimaging.

Conclusions: Chatbot technology could be a valuable tool for medical education and clinical decision support in psychiatry, and as technology continues to advance, these models are likely to play an increasingly integral role in psychiatric practice.

Keywords: Rasch model; chatbots; psychiatry licensing examination.

Figure 1

Person–item map (PKMAP) of ChatGPT-o1-preview. Vertical units in the map represent logits. The mark “XXX” indicates the chatbot’s ability level. Each item in the map corresponds to a question number from the examination, with a “1” or “0” placed after the item number. A “1” indicates that the question was answered correctly and is positioned on the left side of the map, while a “0” indicates that the question was answered incorrectly and is positioned on the right side. The difficulty of each item is also represented by its position along the vertical axis, showing how challenging the question was relative to the chatbot’s ability.