Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Aug;82(4):757-781.
doi: 10.1177/00131644211043207. Epub 2021 Sep 2.

Detecting Differential Rater Functioning in Severity and Centrality: The Dual DRF Facets Model

Kuan-Yu Jin  1 , Thomas Eckes  2
Affiliations

Detecting Differential Rater Functioning in Severity and Centrality: The Dual DRF Facets Model

Kuan-Yu Jin et al. Educ Psychol Meas. 2022 Aug.

Abstract

Performance assessments heavily rely on human ratings. These ratings are typically subject to various forms of error and bias, threatening the assessment outcomes' validity and fairness. Differential rater functioning (DRF) is a special kind of threat to fairness manifesting itself in unwanted interactions between raters and performance- or construct-irrelevant factors (e.g., examinee gender, rater experience, or time of rating). Most DRF studies have focused on whether raters show differential severity toward known groups of examinees. This study expands the DRF framework and investigates the more complex case of dual DRF effects, where DRF is simultaneously present in rater severity and centrality. Adopting a facets modeling approach, we propose the dual DRF model (DDRFM) for detecting and measuring these effects. In two simulation studies, we found that dual DRF effects (a) negatively affected measurement quality and (b) can reliably be detected and compensated under the DDRFM. Using sample data from a large-scale writing assessment (N = 1,323), we demonstrate the practical measurement consequences of the dual DRF effects. Findings have implications for researchers and practitioners assessing the psychometric quality of ratings.

Keywords: Bayesian estimation; MCMC; differential rater functioning; rater bias.

PubMed Disclaimer

Conflict of interest statement

Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Figures

Figure 1.
Figure 1.
Two items with dispersed steps Panel (a) and condensed steps Panel (b).
Figure 2.
Figure 2.
Influence of dual DRF effects on expected scores: (a) DRF-S, (b) DRF-C, and (c) DRF-S & DRF-C.
Figure 3.
Figure 3.
Observed score distributions (percentages) in Simulation Study 1: (a) Subgroup 1, (b) Subgroup 2, (c) Subgroup 3, and (d) Subgroup 4.
Figure 4.
Figure 4.
Test reliability estimates and mean absolute rank change (MARC) for reference (R) and focal group examinees (F) under the RFM and the DDRFM in Simulation Study 1.
Figure 5.
Figure 5.
Relationship between true and estimated examinee proficiency estimates under the RFM and the DDRFM in Simulation Study 1.
Figure 6.
Figure 6.
DRF-S and DRF-C estimates for 29 raters in the TestDaF essay rating study. Note. Hollow and solid circles refer to non-DRF and DRF raters, respectively.
Figure 7.
Figure 7.
Writing proficiency estimates under the RFM and the DDRFM in the essay rating study: (a) θ estimates, (b) Rank-order change for males, and (c) Rank-order change for females.
See this image and copyright information in PMC

References

    1. Beech J. R., Mackintosh I. C. (2005). Do differences in sex hormones affect handwriting style? Evidence from digit ratio and sex role identity as determinants of the sex of handwriting. Personality and Individual Differences, 39(2), 459–468. 10.1016/j.paid.2005.01.024 - DOI
    1. Boulet J. R., McKinley D. W. (2005). Investigating gender-related construct-irrelevant components of scores on the written assessment exercise of a high-stakes certification assessment. Advances in Health Sciences Education, 10(1), 53–63. 10.1007/s10459-004-4297-y - DOI - PubMed
    1. Congdon P. J., McQueen J. (2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37(2), 163–178. 10.1111/j.1745-3984.2000.tb01081.x - DOI
    1. Eckes T. (2005). Examining rater effects in TestDaf writing and speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly, 2(3), 197–221. 10.1207/s15434311laq0203_2 - DOI
    1. Eckes T. (2015). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments (2nd ed.). Peter Lang.

LinkOut - more resources