Evaluation of an OSCE assessment tool for abdominal ultrasound courses

Abstract

Purpose: The purpose of this study was the conception and evaluation of a standardized and reliable assessment tool in the OSCE format to measure the performance and practical skills of abdominal ultrasound users in PGME.

Materials and methods: The design, logistics, pacing and the choice of tested competencies of a rotating OSCE parcours, as well as the options for quality control using detailed checklists versus global rating scales and different approaches to the training of the involved raters are described. Over the last 15 years the parcours has undergone incremental improvement and has been used in final examinations of abdominal ultrasound courses with approximately 5000 medical students and 2000 residents and fellows. For evaluation, all item difficulties and discrimination coefficients of the individual stations and the reliability (Cronbach's alpha) were calculated for the last 626 assessments.

Results: All 14 hands-on stations showed discrimination coefficients from 0.31 to 0.65 (mean 0.48; SD 0.09). The 13 diagram stations showed mean values of 0.50 (SD 0.16). Data analysis revealed mean homogeneous item difficulties of 0.78 (SD 0.02) and 0.62 (SD 0.04), respectively. Cronbach's alpha was 0.69 with five stations and reached values above 0.8 when more than 8 stations are combined in one parcours.

Conclusion: The homogeneous distribution of item difficulties provides an opportunity for designing different OSCE versions with different levels of reliability. Several options to adjust the cut-off values, the choice of the examined contents and factors that influence the examinees' acceptance of this assessment tool for PGME or CME ultrasound courses are discussed. Overall, the values of reliability and accuracy of this assessment tool are high enough to be used also for high-stakes examinations in the field of abdominal ultrasound.