A signal detection-item response theory model for evaluating neuropsychological measures

Abstract

Introduction: Models from signal detection theory are commonly used to score neuropsychological test data, especially tests of recognition memory. Here we show that certain item response theory models can be formulated as signal detection theory models, thus linking two complementary but distinct methodologies. We then use the approach to evaluate the validity (construct representation) of commonly used research measures, demonstrate the impact of conditional error on neuropsychological outcomes, and evaluate measurement bias.

Method: Signal detection-item response theory (SD-IRT) models were fitted to recognition memory data for words, faces, and objects. The sample consisted of U.S. Infantry Marines and Navy Corpsmen participating in the Marine Resiliency Study. Data comprised item responses to the Penn Face Memory Test (PFMT; N = 1,338), Penn Word Memory Test (PWMT; N = 1,331), and Visual Object Learning Test (VOLT; N = 1,249), and self-report of past head injury with loss of consciousness.

Results: SD-IRT models adequately fitted recognition memory item data across all modalities. Error varied systematically with ability estimates, and distributions of residuals from the regression of memory discrimination onto self-report of past head injury were positively skewed towards regions of larger measurement error. Analyses of differential item functioning revealed little evidence of systematic bias by level of education.

Conclusions: SD-IRT models benefit from the measurement rigor of item response theory-which permits the modeling of item difficulty and examinee ability-and from signal detection theory-which provides an interpretive framework encompassing the experimentally validated constructs of memory discrimination and response bias. We used this approach to validate the construct representation of commonly used research measures and to demonstrate how nonoptimized item parameters can lead to erroneous conclusions when interpreting neuropsychological test data. Future work might include the development of computerized adaptive tests and integration with mixture and random-effects models.

Keywords: Assessment; item response theory; neuropsychology; recognition memory; signal detection theory; traumatic brain injury.

Figure 1

Equal variance, signal detection theory model. μ_T = mean of the distribution of familiarity for targets; μ_F = mean of the distribution of familiarity for foils; d′ = μ_T minus μ_F (memory discrimination); C = criterion; C_center = value of the criterion relative to the midpoint between μ_T and μ_F (bias).

Figure 2

Principle factor analysis scree plot for all recognition memory tests.

Figure 3

Standard error of estimate functions for the signal detection-item response theory models. Face = Penn Face Memory Test. Word = Penn Word Memory Test. Object = Visual Object Learning Test.

Figure 4

Regression of estimates of memory discrimination (θ_d′) onto self-report of head injury with loss of consciousness with distributions of residuals. SE_θ = standard error of estimate. Face = Penn Face Memory Test. Word = Penn Word Memory Test. Object = Visual Object Learning Test.

Figure 5

Test response functions allowing item parameters to vary by groups defined by college versus high school education. Face = Penn Face Memory Test. Word = Penn Word Memory Test. Object = Visual Object Learning Test.