Comparing Virtual Reality-Based and Traditional Physical Objective Structured Clinical Examination (OSCE) Stations for Clinical Competency Assessments: Randomized Controlled Trial

Abstract

Background: Objective structured clinical examinations (OSCEs) are a widely recognized and accepted method to assess clinical competencies but are often resource-intensive.

Objective: This study aimed to evaluate the feasibility and effectiveness of a virtual reality (VR)-based station (VRS) compared with a traditional physical station (PHS) in an already established curricular OSCE.

Methods: Fifth-year medical students participated in an OSCE consisting of 10 stations. One of the stations, emergency medicine, was offered in 2 modalities: VRS and PHS. Students were randomly assigned to 1 of the 2 modalities. We used 2 distinct scenarios to prevent content leakage among participants. Student performance and item characteristics were analyzed, comparing the VRS with PHS as well as with 5 other case-based stations. Student perceptions of the VRS were collected through a quantitative and qualitative postexamination online survey, which included a 5-point Likert scale ranging from 1 (minimum) to 5 (maximum), to evaluate the acceptance and usability of the VR system. Organizational and technical feasibility as well as cost-effectiveness were also evaluated.

Results: Following randomization and exclusions of invalid data sets, 57 and 66 participants were assessed for the VRS and PHS, respectively. The feasibility evaluation demonstrated smooth implementation of both VR scenarios (septic and anaphylactic shock) with 93% (53/57) of students using the VR technology without issues. The difficulty levels of the VRS scenarios (septic shock: P=.67; anaphylactic shock: P=.58) were comparable to the average difficulty of all stations (P=.68) and fell within the reference range (0.4-0.8). In contrast, VRS demonstrated above-average values for item discrimination (septic shock: r'=0.40; anaphylactic shock: r'=0.33; overall r'=0.30; with values >0.3 considered good) and discrimination index (septic shock: D=0.25; anaphylactic shock: D=0.26; overall D=0.16, with 0.2-0.3 considered mediocre and <0.2 considered poor). Apart from some hesitancy toward its broader application in future practical assessments (mean 3.07, SD 1.37 for VRS vs mean 3.65, SD 1.18 for PHS; P=.03), there were no other differences in perceptions between VRS and PHS. Thematic analysis highlighted the realistic portrayal of medical emergencies and fair assessment conditions provided by the VRS. Regarding cost-effectiveness, initial development of the VRS can be offset by long-term savings in recurring expenses like standardized patients and consumables.

Conclusions: Integration of the VRS into the current OSCE framework proved feasible both technically and organizationally, even within the strict constraints of short examination phases and schedules. The VRS was accepted and positively received by students across various levels of technological proficiency, including those with no prior VR experience. Notably, the VRS demonstrated comparable or even superior item characteristics, particularly in terms of discrimination power. Although challenges remain, such as technical reliability and some acceptance concerns, VR remains promising in applications of clinical competence assessment.

Keywords: OSCE; VR; acceptance; assessment; clinical competence; comparative study; cost-effectiveness; discrimination power; effectiveness; item characteristics; medical education; medical students; objective structured clinical examination; physical stations; standardized patients; technical feasibility; technological proficiency; virtual reality.

Figure 1

Representative scenes from the (A) virtual reality–based station (VRS) and (B) the physical station (PHS) counterpart.

Figure 2

Layout of the objective structured clinical examination (OSCE) and data collection methods (green) used in the study from both the virtual reality–based station (VRS) and physical station (PHS). The number of participants, along with those excluded (yellow), is indicated. ACLS: advanced cardiac life support; ECG: electrocardiogram.