The ophthalmic clinical evaluation exercise: reliability determination

Abstract

Purpose: Reliable and valid tools must be developed to assess the core residency competencies identified by the Accreditation Council for Graduate Medical Education. The Ophthalmic Clinical Evaluation Exercise (OCEX) is a tool designed to assess the ophthalmology resident's competence in patient care. The OCEX has been shown to have face and content validity. This study will determine the degree to which the OCEX is reliable and has construct validity.

Participants: Ninety-four academic ophthalmology teaching faculty from ophthalmology residency programs across the country.

Methods: Participants reviewed a video compact disc of the same resident and new patient encounter and then completed the OCEX. A scoring rubric was provided.

Results: Results indicate that the OCEX is a reliable tool for faculty to use to assess residency competency. The coefficient alpha statistic (a measure of reliability/internal consistency) for the OCEX as a whole was 0.81. The alpha statistics for 3 of 4 subscales that comprise the OCEX (i.e., interviewing skills = 0.65, interpersonal skills/professionalism = 0.73, case presentation = 0.70) were lower than the OCEX as a whole, but were acceptable for new scales. However, the alpha for the examination subscale (i.e., 0.27) was extremely low. Interrater reliability assessment shows that of 33 individual OCEX items, 31 (94%) had at least 85% of the raters rating the student in 1 of 2 consecutive rating categories.

Conclusions: The OCEX shows both reliability and validity and, therefore, meets the Accreditation Council for Graduate Medical Education criteria for an acceptable assessment tool.