Comparing assessments of students' knowledge by computerized open-ended and multiple-choice tests

Abstract

At the State University of New York at Buffalo School of Medicine and Biomedical Sciences, interactive computerized tests that accept unrestricted natural-language input were used to assess the knowledge of medical students in a sophomore course on clinical biophysics in 1989. The course covered the physical bases, scopes, and limitations of the most widely used diagnostic, therapeutic, and prosthetic technologies. The test was composed of both open-ended sequential questions and multiple-choice questions on the same material. The scores for the two testing modes were correlated, paying special attention to answers that demonstrated a flagrant lack of knowledge, corroborated by subsequent answers. The correlation between the qualities of the answers for the two testing modalities was found to be significantly positive for the lower half of the class of students and negative for the upper half, suggesting that the two modes of testing measure different aspects of competence. The implications of the findings to medical education and accreditation are discussed.