DIF Statistical Inference Without Knowing Anchoring Items

Abstract

Establishing the invariance property of an instrument (e.g., a questionnaire or test) is a key step for establishing its measurement validity. Measurement invariance is typically assessed by differential item functioning (DIF) analysis, i.e., detecting DIF items whose response distribution depends not only on the latent trait measured by the instrument but also on the group membership. DIF analysis is confounded by the group difference in the latent trait distributions. Many DIF analyses require knowing several anchor items that are DIF-free in order to draw inferences on whether each of the rest is a DIF item, where the anchor items are used to identify the latent trait distributions. When no prior information on anchor items is available, or some anchor items are misspecified, item purification methods and regularized estimation methods can be used. The former iteratively purifies the anchor set by a stepwise model selection procedure, and the latter selects the DIF-free items by a LASSO-type regularization approach. Unfortunately, unlike the methods based on a correctly specified anchor set, these methods are not guaranteed to provide valid statistical inference (e.g., confidence intervals and p-values). In this paper, we propose a new method for DIF analysis under a multiple indicators and multiple causes (MIMIC) model for DIF. This method adopts a minimal [Formula: see text] norm condition for identifying the latent trait distributions. Without requiring prior knowledge about an anchor set, it can accurately estimate the DIF effects of individual items and further draw valid statistical inferences for quantifying the uncertainty. Specifically, the inference results allow us to control the type-I error for DIF detection, which may not be possible with item purification and regularized estimation methods. We conduct simulation studies to evaluate the performance of the proposed method and compare it with the anchor-set-based likelihood ratio test approach and the LASSO approach. The proposed method is applied to analysing the three personality scales of the Eysenck personality questionnaire-revised (EPQ-R).

Keywords: confidence interval; differential item functioning; item response theory; least absolute deviations; measurement invariance.

Fig. 1

The path diagram of a MIMIC model for DIF analysis. The subscript i is omitted for simplicity. The dashed lines from x to $Y_j$ indicate the DIF effects.

Fig. 2

Function $h(c)={\sum }_{j=1}^J|{\gamma }_j^{\ast }-a_j^{\ast }c|$, where $J=10$, $a_j^{\ast }=1$ for all j, ${\gamma }_j^{\ast }=0$ and 1 for $j=1,\dots ,8$ and $j=9,10$, respectively. The minimal value of h(c) is achieved when $c=0$.

Algorithm 1

.

Algorithm 2

.

Fig. 3

Scatter plots of the coverage rates of the 95% confidence intervals for ${\gamma }_j^{\ast }$’s. x-axes and y-axes are labelled with item numbers and coverage rates, respectively. Panels a–d correspond to our proposed method, and panels e–h correspond to the Wald intervals constructed with five anchor items. Blue solid circle corresponds to small $d_j$ with high proportion DIF items. Purple solid triangle corresponds to small $d_j$ with medium proportion DIF items. Red solid square corresponds to small $d_j$ with low proportion DIF items. Blue square cross corresponds to large $d_j$ with high proportion DIF items. Purple diamond plus corresponds to large $d_j$ with medium proportion DIF items. Red circle plus corresponds to large $d_j$ with low proportion DIF items.

Fig. 4

Plots of 95% confidence intervals for the DIF parameters ${\gamma }_j^{{\ast }^{\prime }}s$ on scale P, N, and E data sets. The red horizontal lines denote $\gamma =0$. Items are arranged according to the increasing P-values.