The Jack and Jill Adaptive Working Memory Task: Construction, Calibration and Validation

Abstract

Visuospatial working memory (VSWM) is essential to human cognitive abilities and is associated with important life outcomes such as academic performance. Recently, a number of reliable measures of VSWM have been developed to help understand psychological processes and for practical use in education. We sought to extend this work using Item Response Theory (IRT) and Computerised Adaptive Testing (CAT) frameworks to construct, calibrate and validate a new adaptive, computerised, and open-source VSWM test. We aimed to overcome the limitations of previous instruments and provide researchers with a valid and freely available VSWM measurement tool. The Jack and Jill (JaJ) VSWM task was constructed using explanatory item response modelling of data from a sample of the general adult population (Study 1, N = 244) in the UK and US. Subsequently, a static version of the task was tested for validity and reliability using a sample of adults from the UK and Australia (Study 2, N = 148) and a sample of Russian adolescents (Study 3, N = 263). Finally, the adaptive version of the JaJ task was implemented on the basis of the underlying IRT model and evaluated with another sample of Russian adolescents (Study 4, N = 239). JaJ showed sufficient internal consistency and concurrent validity as indicated by significant and substantial correlations with established measures of working memory, spatial ability, non-verbal intelligence, and academic achievement. The findings suggest that JaJ is an efficient and reliable measure of VSWM from adolescent to adult age.

Fig 1. Schematic of the four consecutive screens of a JaJ trial of length two.

Note: Jill always stays in the same position holding a blue ball in her right (from the participant’s perspective) hand (Panels 1–4), while Jack rotates around his axis on each stimulus presentation and can hold a ball in either his right or left hand (Panels 1–2). Jack’s ball also moves, randomly taking one of the 6 marked possible positions on the screen (orange dots). On each stimulus presentation, participants are required to perform two tasks: a) indicate whether Jack holds the ball in the same hand as Jill; and b) memorise the current ball position. At the end of each trial, participants are asked to recall the sequence of the ball positions by clicking on the marked positions in the correct order (Panels 3–4). Cursor locations represent correct answers.

Fig 2. Regression lines representing the decrease in response accuracy with increasing sequence length.

Note: The black line represents proportion of correct responses by length of sequences to be remembered; Error bars represent 95% CI of the proportion based on the standard error. The blue regression line was fit to sequences from length 1 to 7. The red line was only fit to sequences of length 2 to 7. The red regression line fits the empirical average accuracies connected by the dashed line fairly closely, suggesting an approximate linear trend.

Fig 3. Jack and Jill task performance by educational track.

Note: Error bars represent the 95% CI around the mean.

Fig 4. Indicators of correlational test validity, measurement error and reliability by task length.

A. Correlations with other cognitive performance tasks by length of the a-JaJ test. B. Standard measurement error by test length. C. Marginal reliability of test scores by test length.