Implementing a Standardized Effect Size in the POLYSIBTEST Procedure

Abstract

A study was conducted to implement the use of a standardized effect size and corresponding classification guidelines for polytomous data with the POLYSIBTEST procedure and compare those guidelines with prior recommendations. Two simulation studies were included. The first identifies new unstandardized test heuristics for classifying moderate and large differential item functioning (DIF) for polytomous response data with three to seven response options. These are provided for researchers studying polytomous data using POLYSIBTEST software that has been published previously. The second simulation study provides one pair of standardized effect size heuristics that can be employed with items having any number of response options and compares true-positive and false-positive rates for the standardized effect size proposed by Weese with one proposed by Zwick et al. and two unstandardized classification procedures (Gierl; Golia). All four procedures retained false-positive rates generally below the level of significance at both moderate and large DIF levels. However, Weese's standardized effect size was not affected by sample size and provided slightly higher true-positive rates than the Zwick et al. and Golia's recommendations, while flagging substantially fewer items that might be characterized as having negligible DIF when compared with Gierl's suggested criterion. The proposed effect size allows for easier use and interpretation by practitioners as it can be applied to items with any number of response options and is interpreted as a difference in standard deviation units.

Keywords: DIF; POLYSIBTEST; differential item functioning; polytomous data; standardized effect size.

Figure 1.

Relationship Between ${\hat{\mathrm{\beta }}}_{\mathrm{uni}}$ and ${\mathrm{\delta }}_{\mathrm{beta}}$ for Items With Varying Response Categories. ${\hat{\mathrm{\beta }}}_{\mathrm{uni}}=0.100,$ is the value Gierl (2005) recommended to classify a polytomous item with four response categories as having large DIF; ${\mathrm{\delta }}_{\mathrm{beta}}$ values of .164 and .241 are the standardized effect size heuristics that Weese (2020) recommended for classifying moderate and large DIF.

Figure 2.

Overall True-Positive Rates by Number of Response Categories.

Figure 3.

True-Positive Rate by Between-Group Differences in Threshold Parameters for Three to Five Categories.

Figure 4.

True-Positive Rate by Between-Group Differences in Threshold Parameters for Six and Seven Categories.