Evidence of systematic attenuation in the measurement of cognitive deficits in schizophrenia

Abstract

Cognitive tasks that are too hard or too easy produce imprecise measurements of ability, which, in turn, attenuates group differences and can lead to inaccurate conclusions in clinical research. We aimed to illustrate this problem using a popular experimental measure of working memory-the N-back task-and to suggest corrective strategies for measuring working memory and other cognitive deficits in schizophrenia. Samples of undergraduates (n = 42), community controls (n = 25), outpatients with schizophrenia (n = 33), and inpatients with schizophrenia (n = 17) completed the N-back. Predictors of task difficulty-including load, number of word syllables, and presentation time-were experimentally manipulated. Using a methodology that combined techniques from signal detection theory and item response theory, we examined predictors of difficulty and precision on the N-back task. Load and item type were the 2 strongest predictors of difficulty. Measurement precision was associated with ability, and ability varied by group; as a result, patients were measured more precisely than controls. Although difficulty was well matched to the ability levels of impaired examinees, most task conditions were too easy for nonimpaired participants. In a simulation study, N-back tasks primarily consisting of 1- and 2-back load conditions were unreliable, and attenuated effect size (Cohen's d) by as much as 50%. The results suggest that N-back tasks, as commonly designed, may underestimate patients' cognitive deficits as a result of nonoptimized measurement properties. Overall, this cautionary study provides a template for identifying and correcting measurement problems in clinical studies of abnormal cognition. (PsycINFO Database Record

Figure 1

Equal variance, signal detection theory model. μ_T = mean of the distribution of familiarity for targets; μ_F = mean of the distribution of familiarity for foils; d′ = μ_T minus μ_F (discrimination); C = criterion; C_center = value of the criterion relative the midpoint between μ_T and μ_F (bias).

Figure 2

Example of a 2-back run from the N-back task. Examinees are asked to respond whenever a word is repeated from 2 words before. Items repeated from 2-back are targets, items that are repeated, but not from 2-back are referred to as lures, and non-repeated items are referred to as foils.

Figure 3

Item accuracy by group, item type, and N-back. n refers to the number of observed item responses.

Figure 4

Estimates of discriminability (d′) against the measurement error (PSD) of each estimate. PSD = posterior standard deviation. Data for inpatients were omitted because, due to being administered fewer items by design (see methods), PSDs associated with inpatients’ ability estimates are higher than other groups

Figure 5

Information functions for all combinations of N-back load by item type holding all other task factors at their median values. For interpretability, the information functions (represented by solid, dashed, and dotted lines corresponding to 1, 2, and 3-back loads) are superimposed over the implied distributions of discriminability (d′) in undergraduates, community controls, outpatients, and inpatients.