Exploring the potential of artificial intelligence chatbots in prosthodontics education

Abstract

Background: The purpose of this study was to evaluate the performance of widely used artificial intelligence (AI) chatbots in answering prosthodontics questions from the Dentistry Specialization Residency Examination (DSRE).

Methods: A total of 126 DSRE prosthodontics questions were divided into seven subtopics (dental morphology, materials science, fixed dentures, removable partial dentures, complete dentures, occlusion/temporomandibular joint, and dental implantology). Questions were translated into English by the authors, and this version of the questions were asked to five chatbots (ChatGPT-3.5, Gemini Advanced, Claude Pro, Microsoft Copilot, and Perplexity) within a 7-day period. Statistical analyses, including chi-square and z-tests, were performed to compare accuracy rates across the chatbots and subtopics at a significance level of 0.05.

Results: The overall accuracy rates for the chatbots were as follows: Copilot (73%), Gemini (63.5%), ChatGPT-3.5 (61.1%), Claude Pro (57.9%), and Perplexity (54.8%). Copilot significantly outperformed Perplexity (P = 0.035). However, no significant differences in accuracy were found across subtopics among chatbots. Questions on dental implantology had the highest accuracy rate (75%), while questions on removable partial dentures had the lowest (50.8%).

Conclusion: Copilot showed the highest accuracy rate (73%), significantly outperforming Perplexity (54.8%). AI models demonstrate potential as educational support tools but currently face limitations in serving as reliable educational tools across all areas of prosthodontics. Future advancements in AI may lead to better integration and more effective use in dental education.

Keywords: AI chatbot evaluation; Artificial intelligence applications; Clinical decision-support systems; Dentistry specialization; Prosthodontics education.

Fig. 1

Distribution and categorization of prosthodontics questions from the DSRE

Fig. 2

Grouping of questions with simultaneous incorrect answers by AIs: A summary table of the number of incorrect answers to the questions: Level 0 includes questions with no incorrect answers, where all AIs answered correctly simultaneously; Level 1 consists of questions where 1 AI provided an incorrect answer; Level 2 includes questions where 2 AIs provided incorrect answers simultaneously; Level 3 covers questions where 3 AIs provided incorrect answers simultaneously; Level 4 consists of questions where 4 AIs provided incorrect answers simultaneously; and Level 5 includes questions where all 5 AIs provided incorrect answers