Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2022 May;27(2):405-425.
doi: 10.1007/s10459-022-10092-z. Epub 2022 Mar 1.

Feasibility assurance: a review of automatic item generation in medical assessment

Filipe Falcão  1   2 , Patrício Costa  3   4 , José M Pêgo  3   4
Affiliations
Review

Feasibility assurance: a review of automatic item generation in medical assessment

Filipe Falcão et al. Adv Health Sci Educ Theory Pract. 2022 May.

Abstract

Background: Current demand for multiple-choice questions (MCQs) in medical assessment is greater than the supply. Consequently, an urgency for new item development methods arises. Automatic Item Generation (AIG) promises to overcome this burden, generating calibrated items based on the work of computer algorithms. Despite the promising scenario, there is still no evidence to encourage a general application of AIG in medical assessment. It is therefore important to evaluate AIG regarding its feasibility, validity and item quality.

Objective: Provide a narrative review regarding the feasibility, validity and item quality of AIG in medical assessment.

Methods: Electronic databases were searched for peer-reviewed, English language articles published between 2000 and 2021 by means of the terms 'Automatic Item Generation', 'Automated Item Generation', 'AIG', 'medical assessment' and 'medical education'. Reviewers screened 119 records and 13 full texts were checked according to the inclusion criteria. A validity framework was implemented in the included studies to draw conclusions regarding the validity of AIG.

Results: A total of 10 articles were included in the review. Synthesized data suggests that AIG is a valid and feasible method capable of generating high-quality items.

Conclusions: AIG can solve current problems related to item development. It reveals itself as an auspicious next-generation technique for the future of medical assessment, promising several quality items both quickly and economically.

Keywords: Assessment; Automatic item generation; Computer-based testing; Medical Assessment; Multiple-choice questions.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
AIG three-step process for generating medical MCQs
Fig. 2
Fig. 2
Flow chart of the included studies
See this image and copyright information in PMC

References

    1. Arendasy M, Sommer M. Using psychometric technology in educational assessment: The case of a schema-based isomorphic approach to the automatic generation of quantitative reasoning items. Learning and Individual Differences. 2007;17(4):366–383. doi: 10.1016/j.lindif.2007.03.005. - DOI
    1. Baethge C, Goldbeck-Wood S, Mertens S. SANRA—a scale for the quality assessment of narrative review articles. Research Integrity and Peer Review. 2019;4(1):2–8. doi: 10.1186/s41073-019-0064-8. - DOI - PMC - PubMed
    1. Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S. General competencies and accreditation in graduate medical education. Health Affairs. 2002;21(5):103–111. doi: 10.1377/hlthaff.21.5.103. - DOI - PubMed
    1. Blum D, Holling H. Automatic generation of figural analogies with the IMak package. Frontiers in Psychology. 2018;9(AUG):1–13. doi: 10.3389/fpsyg.2018.01286. - DOI - PMC - PubMed
    1. Choi J, Kim H, Pak S. Evaluation of Automatic Item Generation Utilities in Formative Assessment Application for Korean High School Students. Journal of Educational Issues. 2018;4(1):68–89. doi: 10.5296/jei.v4i1.12630. - DOI

LinkOut - more resources