Choosing medical assessments: Does the multiple-choice question make the grade?

Abstract

Background: The multiple-choice question (MCQ) has been shown to measure the same constructs as the short-answer question (SAQ), yet the use of the latter persists. The study aims to evaluate whether assessment using the MCQ alone provides the same outcomes as testing with the SAQ.

Methods: A prospective study design was used. A total of 276 medical students participated in a mock examination consisting of forty MCQs paired to forty SAQs, each pair matched in cognitive skill level and content. Each SAQ was marked by three independent markers. The impact of item-writing flaws (IWFs) on examination outcome was also evaluated.

Results: The intraclass correlation coefficient (ICC) was 0.75 for the year IV examinations and 0.68 for the year V examinations. MCQs were more prone to IWFs than SAQs, but the effect when present in the latter was greater. Removal of questions containing IWFs from the year V SAQ allowed 39% of students who would otherwise have failed to pass.

Discussion: The MCQ can test higher order skills as effectively as the SAQ and can be used as a single format in written assessment provided quality items testing higher order cognitive skills are used. IWFs can have a critical role in determining pass/fail results.

Keywords: Item-writing flaws; inter-rater reliability; medical assessment; multiple-choice question; short-answer question.