Objective skills assessment and construct validation of a virtual reality temporal bone simulator

Abstract

Objectives: To evaluate construct validity of the Voxelman TempoSurg Virtual Reality (VR) temporal bone simulator by determining whether generated objective metrics can distinguish experienced otologic surgeons from intermediate and novice surgeons.

Design: Prospective assessment study.

Setting: Two university-affiliated teaching hospitals.

Participants: Sixty-five participants were recruited; 40 novice surgeons, 15 trainees in otolaryngology, and 10 experienced otolaryngology consultants with a specialist interest in otology were individually assessed on a standardized simulated temporal bone task. The task involved identification and delineation of the sigmoid sinus in a virtual left-sided temporal bone.

Main outcome measures: Objective data were produced using a scoring matrix incorporated into the VOXEL MAN TempoSurg software. The simulator measured the total time taken to complete the task, the volume and efficiency of bone removal and error data for excessive force or injury to the facial nerve, dura, and sigmoid sinus.

Results: Experts and intermediates outperformed novices with respect to the total time taken to complete the task (expert versus novice: p < 0.001; intermediate versus novice: p < 0.001), total volume of bone removed (p < 0.001 and p = 0.03), efficiency of bone removal (p < 0.001 and p < 0.001), time spent with the drill tip obscured (p = 0.002 and p < 0.001), and number of injuries to the sigmoid sinus (p < 0.001 and p < 0.001). The intermediate group injured the sigmoid sinus on more occasions than the experts (p = 0.008) and were less efficient than experienced surgeons (p = 0.005).

Conclusion: Simulator-generated objective metrics can be used to differentiate individuals of differing levels of experience using a standardized temporal bone task. VR simulation has potential as a training tool and may have a role in both formative and summative assessment.