Obstacles Affecting the Implementation of the O-SCORE for Assessment of Orthopedic Surgical Skills Competency

Abstract

Objectives: There is a need for meaningful and reliable measures of surgical competency in residency education. The goal of the current study is to incorporate the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) into the process of resident evaluation at our institution and to assess the feasibility and effectiveness of its use through a web-based platform.

Design: This is a feasibility study that prospectively assesses the implementation of a web-based O-SCORE at our institution. Over a 16-week period, 19 orthopedic surgery residents (PGY2-PGY5) participated in a quality improvement study, which involved collecting 2 feedback forms per week. Each form consisted of a resident form and a linked attending form. At the conclusion of the 16-week trial period, residents and faculty members were asked to complete a survey about their perceptions of the O-SCORE program.

Setting: An academic medical center.

Participants: The study included only residents in postgraduate training years (PGY) 2 through 5 (n = 20) and attendings (n = 37).

Results: During the 16-week study period, 608 resident surveys were requested for the 19 participating residents, of which 404 surveys (66.5%) were completed. Faculty completed 207 of 326 surveys for an overall compliance rate of 63.5%. The O-SCORE was able to significantly differentiate between all training years (p < 0.0001) with the exception of PGY3 residents when compared to PGY4 residents. Overall, residents and faculty found the program valuable and feasible. Resident and faculty perception of the value of the O-SCORE correlated with compliance rate of the O-SCORE surveys.

Conclusions: This study demonstrates that implementation of an immediate feedback program utilizing an electronic platform is achievable and offers reproducible construct validity. However, issues affecting compliance among both residents and faculty physicians must temper optimism for the program and should be systematically addressed to allow for successful implementation.

Keywords: Competency; Education; Interpersonal and Communication Skills; Orthopedic surgery; Ottawa Surgical Competency Operating Room Evaluation; Patient Care; Practice-Based Learning and Improvement; Systems-Based Practice.