Should essays and other "open-ended"-type questions retain a place in written summative assessment in clinical medicine?

Abstract

Background: Written assessments fall into two classes: constructed-response or open-ended questions, such as the essay and a number of variants of the short-answer question, and selected-response or closed-ended questions; typically in the form of multiple-choice. It is widely believed that constructed response written questions test higher order cognitive processes in a manner that multiple-choice questions cannot, and consequently have higher validity.

Discussion: An extensive review of the literature suggests that in summative assessment neither premise is evidence-based. Well-structured open-ended and multiple-choice questions appear equivalent in their ability to assess higher cognitive functions, and performance in multiple-choice assessments may correlate more highly than the open-ended format with competence demonstrated in clinical practice following graduation. Studies of construct validity suggest that both formats measure essentially the same dimension, at least in mathematics, the physical sciences, biology and medicine. The persistence of the open-ended format in summative assessment may be due to the intuitive appeal of the belief that synthesising an answer to an open-ended question must be both more cognitively taxing and similar to actual experience than is selecting a correct response. I suggest that cognitive-constructivist learning theory would predict that a well-constructed context-rich multiple-choice item represents a complex problem-solving exercise which activates a sequence of cognitive processes which closely parallel those required in clinical practice, hence explaining the high validity of the multiple-choice format.

Summary: The evidence does not support the proposition that the open-ended assessment format is superior to the multiple-choice format, at least in exit-level summative assessment, in terms of either its ability to test higher-order cognitive functioning or its validity. This is explicable using a theory of mental models, which might predict that the multiple-choice format will have higher validity, a statement for which some empiric support exists. Given the superior reliability and cost-effectiveness of the multiple-choice format consideration should be given to phasing out open-ended format questions in summative assessment. Whether the same applies to non-exit-level assessment and formative assessment is a question which remains to be answered; particularly in terms of the educational effect of testing, an area which deserves intensive study.

Figure 1

Modified bloom’s taxonomy [ [11]].

Figure 2

Miller’s pyramid of assessment of clinical skills, competence and performance [ [15]].

Figure 3

Stylized depiction of the contrasting ability of the presumed open-ended and multiple-choice formats to assess recognition and recall as opposed to higher forms of cognitive learning. Ideally, multiple-choice and open-ended questions would measure two different abilities (such as recall/recognition versus reasoning/application) – this may be shown as two divergent axes (shown on left). The error variance associated with each type of question is indicated by the shaded blocks, and is much greater for the open-ended question, given its inherent lower reliability. In practice, it appears that the two axes are closely aligned, implying that the two types of questions are measuring essentially the same thing (shown on right). What little additional information the open-ended question might be giving (as shown by a slight divergence in axis) is offset by its wide error variance, which in effect overlaps the information given by the multiple-choice question, thus significantly reducing the value of any additional information it provides.