A comparison of knowledge, synthesis, and clinical judgment. Multiple-choice questions in the assessment of physician competence

Abstract

This study compares the reliability, validity, and efficiency of three multiple-choice question (MCQs) ability scales with patient management problems (PMPs). Data are from the 1980, 1981, and 1982 American Board of Internal Medicine Certifying Examinations. The MCQ ability scales were constructed by classifying the one best answer and multiple-true/false questions in each examination as measuring predominantly clinical judgment, synthesis, or knowledge. Clinical judgment items require prioritizing or weighing management decisions; synthesis items require the integration of findings into a diagnostic decision; and knowledge items stress recall of factual information. Analyses indicate that the MCQ ability scales are more reliable and valid per unit of testing time than are PMPs and that clinical judgment and synthesis scales are slightly more correlated with PMPs than is the knowledge scale. Additionally, all MCQ ability scales seem to be measuring the same aspects of competence as PMPs.