An Experimental Design to Investigate Item Parameter Drift

Abstract

Methods for detecting item parameter drift may be inadequate when every exposed item is at risk for drift. To address this scenario, a strategy for detecting item parameter drift is proposed that uses only unexposed items deployed in a stratified random method within an experimental design. The proposed method is illustrated by investigating unexpected score increases on a high-stakes licensure exam. Results for this example were suggestive of item parameter drift but not significant at the .05 level.

Keywords: differential item functioning; invariance; item parameter drift; test security.

Figure 1.

Form assembly design. Anchor sets are common across years and the study sets are unique but randomly equivalent sets of new (previously unseen) items. To ensure that the study sets are randomly equivalent, both sets must be assembled prior to Year 1.

Figure 2.

Frequency histogram of item difficulty estimates for Year 1 and Year 2 study items. In the figure legend, the difficulty median and standard deviation for each year are also reported along with the difference in medians.

Figure 3.

Frequency histogram of differences in median difficulties across years if the null hypothesis of no IPD is true. This is a randomization distribution based on 1,000,000 permutations of item assignment to year. It can be seen that a value equal or more extreme to the observed value $d*$ is expected to arise due to sampling error 9.8% of the time under the null hypothesis.