Modern artificial intelligence and large language models in graduate medical education: a scoping review of attitudes, applications & practice

Abstract

Background: Artificial intelligence (AI) holds transformative potential for graduate medical education (GME), yet, a comprehensive exploration of AI's applications, perceptions, and limitations in GME is lacking.

Objective: To map the current literature on AI in GME, identifying prevailing perceptions, applications, and research gaps to inform future research, policy discussions, and educational practices through a scoping review.

Methods: Following the Joanna Briggs Institute guidelines and the PRISMA-ScR checklist a comprehensive search of multiple databases up to February 2024 was performed to include studies addressing AI interventions in GME.

Results: Out of 1734 citations, 102 studies met the inclusion criteria, conducted across 16 countries, predominantly from North America (72), Asia (14), and Europe (6). Radiology had the highest number of publications (21), followed by general surgery (11) and emergency medicine (8). The majority of studies were published in 2023. Several key thematic areas emerged from the literature. Initially, perceptions of AI in graduate medical education (GME) were mixed, but have increasingly shifted toward a more favorable outlook, particularly as the benefits of AI integration in education become more apparent. In assessments, AI demonstrated the ability to differentiate between skill levels and offer meaningful feedback. It has also been effective in evaluating narrative comments to assess resident performance. In the domain of recruitment, AI tools have been applied to analyze letters of recommendation, applications, and personal statements, helping identify potential biases and improve equity in candidate selection. Furthermore, large language models consistently outperformed average candidates on board certification and in-training examinations, indicating their potential utility in standardized assessments. Finally, AI tools showed promise in enhancing clinical decision-making by supporting trainees with improved diagnostic accuracy and efficiency.

Conclusions: This scoping review provides a comprehensive overview of applications and limitations of AI in GME but is limited with potential biases, study heterogeneity, and evolving nature of AI.

Keywords: AI integration in medical residency training; AI-driven assessments in medical education; Applications of AI in GME; Artificial intelligence in graduate medical education; Large language models in medical education; Perceptions of AI in medical training.

Fig. 1

Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) flow chart

Fig. 2

Study breakdown based on country where it was done