Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Jun;55(4):2109-2124.
doi: 10.3758/s13428-022-01910-8. Epub 2022 Jul 11.

Comparing the prediction performance of item response theory and machine learning methods on item responses for educational assessments

Jung Yeon Park  1   2 , Klest Dedja  3 , Konstantinos Pliakos  3 , Jinho Kim  4   5 , Sean Joo  2   6 , Frederik Cornillie  2 , Celine Vens  3 , Wim Van den Noortgate  2
Affiliations

Comparing the prediction performance of item response theory and machine learning methods on item responses for educational assessments

Jung Yeon Park et al. Behav Res Methods. 2023 Jun.

Abstract

To obtain more accurate and robust feedback information from the students' assessment outcomes and to communicate it to students and optimize teaching and learning strategies, educational researchers and practitioners must critically reflect on whether the existing methods of data analytics are capable of retrieving the information provided in the database. This study compared and contrasted the prediction performance of an item response theory method, particularly the use of an explanatory item response model (EIRM), and six supervised machine learning (ML) methods for predicting students' item responses in educational assessments, considering student- and item-related background information. Each of seven prediction methods was evaluated through cross-validation approaches under three prediction scenarios: (a) unrealized responses of new students to existing items, (b) unrealized responses of existing students to new items, and (c) missing responses of existing students to existing items. The results of a simulation study and two real-life assessment data examples showed that employing student- and item-related background information in addition to the item response data substantially increases the prediction accuracy for new students or items. We also found that the EIRM is as competitive as the best performing ML methods in predicting the student performance outcomes for the educational assessment datasets.

Keywords: Background information; Educational assessment; Explanatory item response model; Item response theory; Machine learning; Prediction performance.

PubMed Disclaimer

Conflict of interest statement

The authors declared no potential conflicts of interests with respect to the research, authorship and/or publication of this article.

Figures

Fig. 1
Fig. 1
A simple illustration example of a decision tree
Fig. 2
Fig. 2
A schematic illustration of a multi-layer perceptron with input, hidden and output layers
Fig. 3
Fig. 3
Illustrations of the three prediction scenarios
Fig. 4
Fig. 4
An illustration of the data matrix construction for the ML methods
Fig. 5
Fig. 5
Summary of simulation study: AUPR
Fig. 6
Fig. 6
Summary of simulation study: AUROC
Fig. 7
Fig. 7
Summary of simulation study: MSE
Fig. 8
Fig. 8
Results of post hoc tests after Friedman test (AUROC)
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Altman NS. An introduction to kernel and nearest-neighbor nonparametric regression. American Statistician. 1992;46(3):175–185. doi: 10.1080/00031305.1992.10475879. - DOI
    1. Anderson JO, Lin H, Treagust DF, Ross SP, Yore LD. Using large-scale assessment datasets for research in science and mathematics education: Programme for international student assessment (PISA) International Journal of Science and Mathematics Education. 2007;5(4):591–614. doi: 10.1007/s10763-007-9090-y. - DOI
    1. Bates D, Maechler M, Bolker B, Walker S. lme4: Linear mixed-effects models using Eigen and S4. R package version. 2014;1:1–17.
    1. Bergner, Y., Droschler, S., Kortemeyer, G., Rayyan, S., Seaton, D., & Pritchard, D. E. (2012). Model-based collaborative filtering analysis of student response data: Machine-learning item response theory. International Educational Data Mining Society.
    1. Breiman L. Random forests. Machine Learning. 2001;45(1):5–32. doi: 10.1023/A:1010933404324. - DOI

LinkOut - more resources