Investigation of Objectivity in Scoring and Evaluating Microvascular Anastomosis Simulation Training

Abstract

The increase in minimally invasive surgery has led to a decrease in surgical experience. To date, there is only limited research examining whether skills are evaluated objectively and equally in simulation training, especially in microsurgery. The purpose of this study was to analyze the objectivity and equality of simulation evaluation results conducted in a contest format. A nationwide recruitment process was conducted to select study participants. Participants were recruited from a pool of qualified physicians with less than 10 years of experience. In this study, the simulation procedure consisted of incising a 1 mm thick blood vessel and suturing it with a 10-0 thread using a microscope. Initially, we planned to have the neurosurgical supervisors score the simulation procedure by direct observation. However, due to COVID-19, some study participants were unable to attend. Thus requiring some simulation procedures to be scored by video review. A total of 14 trainees participated in the study. The Cronbach's alpha coefficient among the scorers was 0.99, indicating a strong correlation. There was no statistically significant difference between the scores from the video review and direct observation judgments. There was a statistically significant difference (p <0.001) between the scores for some criteria. For the eight criteria, individual scorers assigned scores in a consistent pattern. However, this pattern differed between scorers indicating that some scorers were more lenient than others. The results indicate that both video review and direct observation methods are highly objective techniques evaluate simulation procedures.

Keywords: microsurgery; objective assessment; skills; techniques.

Fig. 1

The review video consisted of a video of the microscope screen and a video of the surgeon’s entire body taken from the side and back of the surgeon.

Fig. 2

A view of the venue at the actual contest.