Novel hybrid objective structured assessment of technical skills/objective structured clinical examinations in comprehensive perioperative breast care: a three-year analysis of outcomes

Abstract

Purpose: Objective Structured Assessment of Technical Skills (OSATS) and Objective Structured Clinical Examinations (OSCE) are common tools used to objectively evaluate surgical residents. In 2005, our institution presented a novel hybrid OSATS/OSCE, which we renamed the Objective Structured Clinical Assessment (OSCA), encompassing all 6 core competencies regarding comprehensive care of the breast care patient. This study presents an analysis of the effects of a compulsory, comprehensive OSCA on our residents' competence in this index learning category.

Methods: Completed breast OSCA, Accreditation Council for Graduate Medical Education (ACGME) operative logs, and Report D of the American Board of Surgery in Training Examination (ABSITE) were collected for 24 senior-level residents. The OSCA was implemented for post graduate year (PGY)-3 residents in the 2003-2004 year. Data were also collected for an equal number of residents before the OSCA. Competence of residents was evaluated using the procedure-based OSCA that encompasses the 6 core competencies, completed for each resident by a specialty breast surgeon. Next, ACGME operative logs for graduating seniors were analyzed. Finally, breast care questions were identified from Report D. The number of residents scoring incorrectly on these questions was divided by the total number of resident-questions to yield a percent incorrect for each group. Similarly, the percent of total incorrect ABSITE examination questions was calculated, assessing overall group improvement.

Results: All assigned residents achieved competence in all designated categories of the OSCA. Since the OSCA, the average number of breast cases in our program has risen from a mean of 102 cases to 124 cases per graduating senior. ABSITE data showed 264 resident-questions before the OSCA, with 26.5% incorrect responses. After OSCA, there were 252 resident-questions, with 18.25% incorrect. This represents a significant decrease (p < 0.05) of incorrect responses to breast questions on the ABSITE. Before the OSCA, there were 2139 total resident-questions with 25.28% wrong responses and after, there were 2111 resident-questions with 23.97% wrong responses, representing no significant difference (p > 0.05) in incorrect responses on the entire examination.

Conclusions: Our data show consistent competence of residents in breast disease as evaluated by the OSCA, an increase in numbers of breast cases, and a decrease in incorrect responses on breast-related ABSITE questions. We believe a comprehensive, complete care OSCA represents a valuable learning tool for residents to increase their competence and improve their outcomes in breast care. We believe that comprehensive OSCAs will be necessary tools to evaluate resident competence and should be implemented in all areas of general surgery.