Validation of an objective structured assessment of technical skill for surgical residents

Abstract

Purpose: This study examined the concurrent validity of the Objective Structured Assessment of Technical Skill (OSATS), a new test of technical skill for general surgery residents.

Method: Twelve residents (six in their senior, or fifth, year and six in their junior, or third, year) at the University of Toronto in 1994-95 were ranked within level of training according to their OSATS marks and by surgical faculty. Correspondence between OSATS and faculty rankings was assessed using Spearman rank-order correlation coefficients.

Results: The correlations between test scores and faculty rankings were generally high for the senior residents but low for the junior residents.

Conclusion: Scores on the OSATS accurately reflect the independent opinions of faculty regarding the technical skills of senior residents, suggesting that it is a valid measure of technical skill for these individuals. The scores did not, however, reproduce faculty rankings of the junior residents. Whether this was a failing of the OSATS or the faculty rankings requires further study.