Adaptive measurement of cognitive function based on multidimensional item response theory

Abstract

Introduction: Up to 20% of older adults in the United States have mild cognitive impairment (MCI), and about one-third of people with MCI are predicted to transition to Alzheimer's disease (AD) within 5 years. Standard cognitive assessments are long and require a trained technician to administer. We developed the first computerized adaptive test (CAT) based on multidimensional item response theory (MIRT) to more precisely, rapidly, and repeatedly assesses cognitive abilities across the adult lifespan. We present results for a prototype CAT (pCAT-COG) for assessment of global cognitive function.

Methods: We sampled items across five cognitive domains central to neuropsychological testing (episodic memory [EM], semantic memory/language [SM], working memory [WM], executive function/flexible thinking, and processing speed [PS]). The item bank consists of 54 items, with 9 items of varying difficulty drawn from six different cognitive tasks. Each of the 54 items has 3 response trials, yielding an ordinal score (0-3 trials correct). We also include three long-term memory items not designed for adaptive administration, for a total bank of 57 items. Calibration data were collected in-person and online, calibrated using a bifactor MIRT model, and pCAT-COG scores validated against a technician-administered neuropsychological battery.

Results: The bifactor MIRT model improved fit over a unidimensional IRT model (p < 0.0001). The global pCAT-COG scores were inversely correlated with age (r = -0.44, p < 0.0001). Simulated adaptive administration of 11 items maintained a correlation of r = 0.94 with the total item bank scores. Significant differences between mild and no cognitive impairment (NCI) were found (effect size of 1.08 SD units). The pCAT-COG correlated with clinician-based global measure (r = 0.64).

Discussion: MIRT-based CAT is feasible and valid for the assessment of global cognitive impairment, laying the foundation for the development of a full CAT-COG that will draw from a much larger item bank with both global and domain specific measures of cognitive impairment.

Highlights: As Americans age, numbers at risk for developing cognitive impairment are increasing.Aging-related declines in cognition begins decades prior to the onset of obvious cognitive impairment.Traditional assessment is burdensome and requires trained clinicians.We developed an adaptive testing framework using multidimensional item response theory.It is comparable to lengthier in-person assessments that require trained psychometrists.

Keywords: Alzheimer's disease; bifactor model; cognitive impairment; computerized adaptive testing; multidimensional item response theory; neuropsychological assessment.

FIGURE 1

Bifactor model observed versus expected marginal item category proportions.

FIGURE 2

Relationship between global pCAT‐COG score and age. pCAT‐COG, prototype computerized adaptive test.

FIGURE 3

pCAT‐COG global cognition versus RADC composite measures. pCAT‐COG, prototype computerized adaptive test; RADC, Rush Alzheimer's Disease Center.

FIGURE 4

Binary testlet versus ordinal score (0–3 correct) – Stroop.