Explaining Person-by-Item Responses using Person- and Item-Level Predictors via Random Forests and Interpretable Machine Learning in Explanatory Item Response Models

Abstract

This study incorporates a random forest (RF) approach to probe complex interactions and nonlinearity among predictors into an item response model with the goal of using a hybrid approach to outperform either an RF or explanatory item response model (EIRM) only in explaining item responses. In the specified model, called EIRM-RF, predicted values using RF are added as a predictor in EIRM to model the nonlinear and interaction effects of person- and item-level predictors in person-by-item response data, while accounting for random effects over persons and items. The results of the EIRM-RF are probed with interpretable machine learning (ML) methods, including feature importance measures, partial dependence plots, accumulated local effect plots, and the H-statistic. The EIRM-RF and the interpretable methods are illustrated using an empirical data set to explain differences in reading comprehension in digital versus paper mediums, and the results of EIRM-RF are compared with those of EIRM and RF to show empirical differences in modeling the effects of predictors and random effects among EIRM, RF, and EIRM-RF. In addition, simulation studies are conducted to compare model accuracy among the three models and to evaluate the performance of interpretable ML methods.

Keywords: explanatory item response theory; interpretable machine learning; mixed-effects machine learning; random forests.

Figure 1

Empirical study: feature importance measures for EIRM-RF. Note: In the predictor (feature) name, “nom” indicates a nominal predictor, and “c” indicates the mean-centered predictor.

Figure 2

Empirical study: partial dependence plots from EIRM-RF. Note: Continuous predictors were mean-centered; each tick mark in the x-axis represents values of a continuous predictor.

Figure 3

Empirical study: partial dependence plots from EIRM-RF. Note: Continuous predictors were mean-centered; each tick mark in the x-axis represents values of a continuous predictor.

Figure 4

Empirical study: partial dependence plots of EIRM-RF. Note: Continuous predictors were mean-centered; each tick mark in the x-axis represents values of a continuous predictor.

Figure 5

Empirical study: accumulated local plots of EIRM-RF. Note: Continuous predictors were mean-centered; each tick mark in the x-axis represents values of a continuous predictor.

Figure 6

Empirical study: the H-statistic of all predictors from EIRM-RF. Note: In the predictor (feature) name, “nom” indicates a nominal predictor, and “c” indicates the mean-centered predictor; Values on the x-axis indicate the strengths of the interaction effects.

Figure 7

Empirical study: the H-statistic of the two-way interactions between the item format and all other predictors from EIRM-RF. Note: In the predictor (feature) name, “nom” indicates a nominal predictor, and “c” indicates the mean-centered predictor; Values on the x-axis indicate the strengths of the interaction effects.

Figure A.1

Visualization of a generated tree structure.

Figure A2

Selected plots of ‘true’ partial dependence (top) and accumulated local effects (bottom). Note: Continuous predictors were mean-centered; To present patterns in partial dependence and accumulated local effects, the x- and y-axes of the two selected predictors were not displayed on the same scale.

Figure A3

Visualization of a generated tree structure.

Figure A4

Selected plots of “true” partial dependence (top) and accumulated local effects (bottom). Note: Continuous predictors were mean-centered; To present patterns in partial dependence and accumulated local effects, the x- and y-axes of the two selected predictors were not displayed on the same scale.