Measuring the Quality of the OSCE in a Chiropractic Programme: A Review of Metrics and Recommendations

Abstract

Objectives: The Objective Structured Clinical Examination (OSCE) is a commonly used assessment of clinical skill, and ensuring the quality and reliability in OSCEs is a complex, and on-going process. This paper discusses scoring schemas and reviews checklists and global rating scales (GRS) for marking. Also detailed are post-examination quality assurance metrics tailored to smaller cohorts, with an illustrative dataset.

Methods: A deidentified OSCE dataset, from stations with both a checklist and GRS, of 24 examinees from a 2021 cohort was assessed using the following metrics: Cut-scores or pass-rates, number-of-failures, R2, inter-grade discrimination, and between-group-variation. The results were used to inform a set of implementable recommendations to improve future OSCEs.

Results: For most stations the calculated cut-score calculated was higher than the traditional pass of 50% (58.9.8-68.4%). Number-of-failures were low for traditional pass rates and cuts-scores (0.00-16.7%), excepting Lab Analysis where number-of-failures was 50.0%. R2 values ranged from 0.67-0.97, but proportion of total variance was high (67.3-95.9). These data suggest there were potential missed teaching concepts, that station marking was open to examiner interpretation, and there were inconsistencies in examiner marking.Recommendations included increasing checklist detail and using a weighted marking scale, separating some stations into dichotomous and key-feature checklists, using GRSs specific to each station, and reviewing all future OSCEs with the metrics described to guide refinements.

Conclusions: The analysis used revealed several potential issues with the OSCE assessment. These findings informed recommendations to improve the quality of future examinations.

Keywords: Benchmarking; Checklist; Clinical competence; Reproducibility of results.

Figure 1

Plot of mock OSCE scores against their associated mock GRS (1 = Fail, 2 = Pass, 3 = Excellent pass) indicating the difference between a 50% pass (black dashed line) and the cut score (red dashed line), or passing level suggested by a borderline regression analysis.

Figure 2

Plots of all OSCE stations with borderline regression analysis. Panel A shows the Technique 1 (Tech 1) station, panel B the Technique 2 (Tech 2), panel C the Professionalism station, panel D shows Neurology, panel E the Lab analysis station, panel F shows the Systems review (Systems) station, and panel G shows the Care plan station. BLR = borderline regression.