Resident Training and the Assessment of Orthopaedic Surgical Skills

Abstract

Medical knowledge and technical skills are foundations of surgical competency. The American Board of Orthopaedic Surgery (ABOS) and the Resident Review Committee for Orthopaedic Surgery recently mandated simulation training to improve surgical skills, listing 17 surgical skills modules to improve residents' technical skills. However, there is no established tool to measure the effectiveness of these modules. The Global Index for Technical Skills (GRITS) tool has been previously validated for evaluating general surgery residents. The aim of this study was to determine whether the GRITS tool is valid, practical, and reliable in evaluating the skills of orthopaedic residents in a simulation setting, whether the outcomes correlate to performance in the operating room, and to what extent these simulation modules are valued by residents.

Methods: Simulation performance was assessed longitudinally on 5 residents using the GRITS assessment through postgraduate years (PGY) 1 to 5 (n = 25 evaluations) in a simulated volar forearm approach using cadaveric specimens. An additional 20 PGY-1 residents were evaluated cross-sectionally in this same time frame. Written, open-ended feedback on the simulation experience was sought and analyzed via a thematic analysis. For correlative data, evaluations (n = 65 evaluations) of a variety of authentic surgical procedures were compiled on PGY-2 through PGY-5 orthopaedic residents and compared with the simulated experiences.

Results: GRITS scores were averaged for each group of residents, and validity and reliability were assessed using R-software. PGY-1 residents' mean GRITS evaluation score (expressed as a value from 1 to 5) was 3.4. Longitudinally, this mean score increased over the PGY years 2-5 to 4.4, 4.7, 4.9, and 4.8, respectively. Of the parameters measured by GRITS, the lowest average scores were "flow of operation" and "time and motion" across all levels, although these did improve over PGY years 2 to 5. Findings were consistent between simulation and "real-world" procedures. Open-ended responses evaluating the module were positive.

Conclusions: Our study suggests that the GRITS tool shows promise as an effective and reliable method for assessing orthopaedic resident's technical skills based on an ABOS module system.

Fig. 1

Global Rating Index for Technical Skills. (Reproduced, with permission, from: Doyle JD, Webber EM, Siddu RS. A universal global rating skill for the evaluation of technical skills in the operating room. Am J Surg. 2007;193[5]:551-5.)

Fig. 2

Mean scores of evaluations by the level of training (postgraduate year) of simulation surgical procedures. Possible scores range from 1 to 5. Error bars represent SD.

Fig. 3

Mean scores of evaluations by the level of training (postgraduate year) of real surgical procedures. Possible scores range from 1 to 5. Error bars represent SD.