Iteratively refined ChatGPT outperforms clinical mentors in generating high-quality interprofessional education clinical scenarios: a comparative study

Abstract

Background: Interprofessional education (IPE) is essential for promoting teamwork among healthcare professionals. However, its implementation is often hindered by the limited availability of interprofessional faculty and scheduling challenges in creating high-quality IPE scenarios. While AI tools like ChatGPT are increasingly being explored for this purpose, they have yet to demonstrate the ability to generate high-quality IPE scenarios, which remains a significant challenge. This study examines the effectiveness of GPT-4o, an advanced version of ChatGPT enhanced by novel methodologies, in overcoming these obstacles.

Methods: This comparative study assessed clinical scenarios generated by GPT-4o using two strategies-standard prompt (a single-step scenario generation without iterative feedback) and iterative refinement (a multi-step, feedback-driven process)-against those crafted by clinical mentors. The iterative refinement method, inspired by actual clinical scenario development, employs a cyclical process of evaluation and refinement, closely mimicking discussions among professionals. Scenarios were evaluated for time efficiency and quality using the Interprofessional Quality Score (IQS), defined as the mean score assigned by multidisciplinary evaluators across five interprofessional criteria: clinical authenticity, team collaboration, educational alignment, appropriate challenge, and student engagement.

Results: Scenarios developed using the iterative refinement strategy were completed significantly faster than those by clinical mentors and achieved higher or equivalent IQS. Notably, these scenarios matched or exceeded the quality of those created by humans, particularly in areas such as appropriate challenge and student engagement. Conversely, scenarios generated via the standard prompt method exhibited lower accuracy and various other deficiencies. Blinded attribution assessments by students further demonstrated that scenarios developed through iterative refinement were often indistinguishable from those created by human mentors.

Conclusions: Employing GPT-4o with iterative refinement and role-playing strategies produces clinical scenarios that, in some areas, exceed those developed by clinical mentors. This approach reduces the need for extensive faculty involvement, highlighting AI's potential to closely align with established educational frameworks and substantially enhance IPE, particularly in resource-constrained settings.

Keywords: Artificial intelligence; ChatGPT; Clinical scenarios; Interprofessional education; Iterative refinement.

Fig. 1

Schematic representation of the study design

Fig. 2

Time and quality comparison results of clinical scenarios generated by GPT-4o under different strategies versus human creation. a) Average time spent by different groups to create clinical scenarios, with the time for the clinical mentors group representing the cumulative effort of all involved multidisciplinary mentors. b) Comparison of IQS results across various metrics for clinical scenarios created by different methods. Statistical significance was determined using two-tailed T-tests, with * indicating p <.05, ** indicating p <.01 and *** indicating p <.001