Competency domains in an undergraduate Objective Structured Clinical Examination: their impact on compensatory standard setting

Abstract

Context: Following a 15-week attachment in paediatrics and child health, general practice and dermatology medical students in their second clinical year at this medical school undertake a high-stakes assessment including an objective structured clinical examination (OSCE). There were 2 hypotheses. Firstly, groups of similar stations map to competency domains identifiable by factor analysis. Secondly, poor performance in individual domains is compensated for by achieving the required standard of performance across the whole assessment.

Methods: A total of 647 medical students were assessed by an OSCE during 5 individual examination sittings (diets) over 2 years. Ten scoring stations in the OSCE were analysed and confirmatory factor analysis performed comparing a 1-factor model (where all the stations are discrete entities related to one underlying domain) with a 3-factor model (where the stations load onto 3 domains from a previously reported exploratory factor analysis).

Results: The 3-factor model yielded a significantly better fit to the data (chi(2 )=( )15.3, P < 0.01). Assessing the compensation data of 1 diet, 29 of 127 students failed in 1 or more domains described, whereas only 5 failed if compensation was allowed across all domains.

Discussion: Confirmatory factor analysis showed a significant fit of the data to previously described competency domains for a high-stakes undergraduate OSCE. Compensation within but not between competency domains would provide a more robust standard, improve validity, and substantially reduce the pass rate.