Development, Validation, and Reliability of a P1 Objective Structured Clinical Examination Assessing the National EPAs

Abstract

Objective: To document the performance of first-year pharmacy students on a revised objective structured clinical examination (OSCE) based on national entrustable professional activities, identify risk factors for poor performance, and assess its validity and reliability.

Methods: A working group developed the OSCE to verify students' progress toward readiness for advanced pharmacy practice experiences at the L1 level of entrustment (ready for thoughtful observation) on the national entrustable professional activities, with stations cross-mapped to the Accreditation Council for Pharmacy Education educational outcomes. Baseline characteristics and academic performance were used to investigate risk factors for poor performance and validity, respectively, by comparing students who were successful on the first attempt with those who were not. Reliability was evaluated using re-grading by a blinded, independent grader, and analyzed using Cohen's kappa.

Results: A total of 65 students completed the OSCE. Of these, 33 (50.8%) successfully completed all stations on first attempt, and 32 (49.2%) had to re-attempt at least 1 station. Successful students had higher Health Sciences Reasoning Test scores (mean difference 5, 95% CI 2-9). First professional year grade point average was higher for students who passed all stations on first attempt (mean difference 0.4 on a 4-point scale, 95% CI 0.1-0.7). When evaluated in a multiple logistic regression, no differences were statistically significant between groups. Most kappa values were above 0.4 (range 0.404-0.708), suggesting moderate to substantial reliability.

Conclusion: Though predictors of poor performance were not identified when accounting for covariates, the OSCE was found to have good validity and reliability.

Keywords: Entrustable professional activity; Objective structured clinical examination.