Latent Variable Modeling and Adaptive Testing for Experimental Cognitive Psychopathology Research

Abstract

The adaptation of experimental cognitive tasks into measures that can be used to quantify neurocognitive outcomes in translational studies and clinical trials has become a key component of the strategy to address psychiatric and neurological disorders. Unfortunately, while most experimental cognitive tests have strong theoretical bases, they can have poor psychometric properties, leaving them vulnerable to measurement challenges that undermine their use in applied settings. Item response theory-based computerized adaptive testing has been proposed as a solution but has been limited in experimental and translational research due to its large sample requirements. We present a generalized latent variable model that, when combined with strong parametric assumptions based on mathematical cognitive models, permits the use of adaptive testing without large samples or the need to precalibrate item parameters. The approach is demonstrated using data from a common measure of working memory-the N-back task-collected across a diverse sample of participants. After evaluating dimensionality and model fit, we conducted a simulation study to compare adaptive versus nonadaptive testing. Computerized adaptive testing either made the task 36% more efficient or score estimates 23% more precise, when compared to nonadaptive testing. This proof-of-concept study demonstrates that latent variable modeling and adaptive testing can be used in experimental cognitive testing even with relatively small samples. Adaptive testing has the potential to improve the impact and replicability of findings from translational studies and clinical trials that use experimental cognitive tasks as outcome measures.

Keywords: cognitive assessment; cognitive psychometrics; computerized adaptive testing; experimental cognitive psychopathology; item response theory; neurocognitive disorders.

Figure 1.

Hierarchical model depicting three nested levels of variability that are relevant to experimental cognitive testing. Note. Levels are defined by variation over items (Level 1), variation over experimental conditions (Level 2), and variation over examinees (Level 3). Individual item responses are for the ith item within the jth condition for the kth examinee. Examinee parameters vary over all individual examinees and condition parameters vary over all conditions within each examinee. Examinee effects are defined with regard to the experimental manipulation of interest. Thus, the modeling approach separates measures of individual differences among examinees into components that are explained by the experimental manipulation (ω) and components that are not (ζ). The notation y^* reflects that items may be related to the linear model via a link function, thereby permitting item responses that do not have a normal distribution (e.g., dichotomous accuracy scores). Subscript notation in the figure is used to denote variation, not matrix dimensions.

Figure 2.

Equal variance, signal detection theory model. Note. μ_T = mean of the distribution of familiarity for targets; μ_F = mean of the distribution of familiarity for foils; d′ = μ_T minus μ_F (discrimination); C = criterion; C_c = value of the criterion relative the midpoint between μ_T and μ_F (bias).

Figure 3.

Example of the signal detection theory model applied to N-back data within the generalized latent variable model framework proposed Note. This toy model is for a single examinee and a test comprising 1 foil item and 1 target item administered over 1- and 2-back load conditions (i.e., 1B, F = 1-back foil; 1B, T = 1-back target; 2B, F = 2-back foil; 2B, T = 2-back target). The latent response variables (y^*) are regressed onto the latent subject variables representing the subject-level effects of the experimental manipulation (ω) and the residual, condition-level prediction errors (ζ). Γ contains the experimental structure parameters that represent the hypotheses of the investigator. Here, we use orthogonal polynomial contrasts for the two levels of load. The intercept parameters for both d′ and C_c (I_d′ and I_Cc) are multiplied by 1.0 and the slope parameters for both d′ and C_c (S_d′ and S_Cc) are multiped by −1.0 at 1-back and 1.0 at 2-back. The elements of Λ are fixed based on assumptions from the signal detection theory model. Specifically, d′ parameters are weighted by 0.5 for all items and C_c are weighted by −1.0 for target items but 1.0 for foil items.

Figure 4.

Proportion correct by group, N-back condition, and item type