Reliability and Validity of Ambulatory Cognitive Assessments

Abstract

Mobile technologies are increasingly used to measure cognitive function outside of traditional clinic and laboratory settings. Although ambulatory assessments of cognitive function conducted in people's natural environments offer potential advantages over traditional assessment approaches, the psychometrics of cognitive assessment procedures have been understudied. We evaluated the reliability and construct validity of ambulatory assessments of working memory and perceptual speed administered via smartphones as part of an ecological momentary assessment protocol in a diverse adult sample ( N = 219). Results indicated excellent between-person reliability (≥0.97) for average scores, and evidence of reliable within-person variability across measurement occasions (0.41-0.53). The ambulatory tasks also exhibited construct validity, as evidence by their loadings on working memory and perceptual speed factors defined by the in-lab assessments. Our findings demonstrate that averaging across brief cognitive assessments made in uncontrolled naturalistic settings provide measurements that are comparable in reliability to assessments made in controlled laboratory environments.

Keywords: ambulatory assessment; cognition; ecological validity; smartphone.

Figure 1

Example of symbol search test. Participants were asked to decide which of the two pairs at the bottom of the screen was among the pairs at the top of the screen as quickly as possible.

Figure 2

Example of dot memory test. (a) The participants were asked to remember the location of three red dots. (b) After grid was removed, participants were required to touch F’s among E’s for 8 seconds. (c) Empty 5 × 5 grid reappeared on the screen and participants were prompted to recall the location of initial 3 dots.

Figure 3

Example of N-back test. (a) In the practice session, participants were asked to determine whether the two cards matched. (b) Cards shifted position from right to left, with each new target card turning face down. (c) Participants were asked to determine if the current target card matched the face-down card they saw 2 trials back.

Figure 4

The between-person reliability of average scores obtained on each of the days of ambulatory assessments.

Figure 5

The within-person reliability estimated separately for each of the 14 days of ambulatory assessments.

Figure 6

Confirmatory Factor Model and standard coefficients. Ambulatory tasks (dot memory, n-back, and symbol search) are indicated by rectangles with thick borders.