Reliability of multi-station interviews in selection of junior doctors for specialty training

Abstract

Background: Interviews for junior doctor training posts in the UK are considered high stakes assessments following the introduction of Modernizing Medical Careers.

Aims: To examine the reliability of paediatric recruitment in our Deanery for specialty training (ST) in 2008.

Methods: A total of 225 candidates were interviewed for 88 posts. There were three interview stations (presentation, structured interview and communication) each lasting for 10 min. Two interviewers independently marked candidates at each station on a 5 point Likert scale. Interview scores were analysed to evaluate reliability.

Results: Reliability for ST1 interviews was excellent, reasonable for ST2 and ST3 and poor for ST4 (Cronbach's alpha of 0.908, 0.749, 0.759 and 0.648). Generalizability coefficient was good for ST1 interviews (>0.8) but for ST2-ST4 interviews was below acceptable (<0.8). Decision (D) study calculations showed up to 26 stations that were required to achieve minimum reliability.

Conclusions: This three station interview process was not reliable enough for a high stakes assessment. An increased number of stations are required to improve reliability. Further work is required to clarify the optimum number of stations to achieve minimum reliability and feasibility.