Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2006 Sep 14:6:46.
doi: 10.1186/1472-6920-6-46.

Standard setting: comparison of two methods

Sanju George  1 , M Sayeed Haque, Femi Oyebode
Affiliations
Comparative Study

Standard setting: comparison of two methods

Sanju George et al. BMC Med Educ. .

Abstract

Background: The outcome of assessments is determined by the standard-setting method used. There is a wide range of standard-setting methods and the two used most extensively in undergraduate medical education in the UK are the norm-reference and the criterion-reference methods. The aims of the study were to compare these two standard-setting methods for a multiple-choice question examination and to estimate the test-retest and inter-rater reliability of the modified Angoff method.

Methods: The norm-reference method of standard-setting (mean minus 1 SD) was applied to the 'raw' scores of 78 4th-year medical students on a multiple-choice examination (MCQ). Two panels of raters also set the standard using the modified Angoff method for the same multiple-choice question paper on two occasions (6 months apart). We compared the pass/fail rates derived from the norm reference and the Angoff methods and also assessed the test-retest and inter-rater reliability of the modified Angoff method.

Results: The pass rate with the norm-reference method was 85% (66/78) and that by the Angoff method was 100% (78 out of 78). The percentage agreement between Angoff method and norm-reference was 78% (95% CI 69% - 87%). The modified Angoff method had an inter-rater reliability of 0.81-0.82 and a test-retest reliability of 0.59-0.74.

Conclusion: There were significant differences in the outcomes of these two standard-setting methods, as shown by the difference in the proportion of candidates that passed and failed the assessment. The modified Angoff method was found to have good inter-rater reliability and moderate test-retest reliability.

PubMed Disclaimer

References

    1. Kane M. Validating the performance standards associated with passing scores. Rev Educ Res. 1994;64:425–61. doi: 10.2307/1170678. - DOI
    1. Norcini JJ. Setting standards on educational tests. Medical Education. 2003;37:464–469. doi: 10.1046/j.1365-2923.2003.01495.x. - DOI - PubMed
    1. Boursicot K, Roberts T. Setting standards in a professional higher education course: Defining the concept of the minimally competent student in performance based assessment at the level of graduation from medical school. Higher Education Quarterly. 2006;60:74–90. doi: 10.1111/j.1468-2273.2006.00308.x. - DOI
    1. Searle J. Defining competency-the role of standard setting. Medical Education. 2000;34:363–366. doi: 10.1046/j.1365-2923.2000.00690.x. - DOI - PubMed
    1. Case SM, Swanson DB. Constructing written test questions for the basic and clinical sciences. Philadelphia: National Board of Medical Examiners; 1998.

LinkOut - more resources