Reliability and validity of the objective structured clinical examination in assessing surgical residents

Abstract

The purpose of this research was to assess reliability and construct validity of the objective structured clinical examination (OSCE) for evaluating the clinical skills of surgical residents. Reliability refers to precision of the examination and construct validity to the degree to which the examination can discriminate between different levels of training. Twenty-seven second postgraduate year surgical residents took a 38-station OSCE representing seven surgical specialties and that tested history-taking, physical examination, problem-solving, technical skills, and attitudes. A couplet methodology was used wherein a patient encounter was followed by written questions aimed at testing problem-solving and patient management capabilities. Thirty-six standardized patients were trained and 36 surgeons served as examiners marking from structured checklists. Overall reliability, Cronbach's alpha, was 0.89. Construct validity was examined by comparing the scores of the residents with those of a group of graduates of foreign medical schools applying for a "pre-internship" program. For 17 of 19 stations that both groups took, the residents performed significantly better (p less than 0.01). Individual station validity was significant for 32 of 38 stations (r = 0.36 to 0.82, p less than 0.05). The examinations took 3.83 hours at a cost of $5,293 (Canadian dollars). The OSCE has been shown to be a reliable method of assessing clinical skills of surgical residents, construct validity has been established, and inter-item validity confirmed. Reliabilities achieved exceed those traditionally required for both acceptance and promotion decisions.