Reliability and Validity of Smartphone Cognitive Testing for Frontotemporal Lobar Degeneration

Abstract

Importance: Frontotemporal lobar degeneration (FTLD) is relatively rare, behavioral and motor symptoms increase travel burden, and standard neuropsychological tests are not sensitive to early-stage disease. Remote smartphone-based cognitive assessments could mitigate these barriers to trial recruitment and success, but no such tools are validated for FTLD.

Objective: To evaluate the reliability and validity of smartphone-based cognitive measures for remote FTLD evaluations.

Design, setting, and participants: In this cohort study conducted from January 10, 2019, to July 31, 2023, controls and participants with FTLD performed smartphone application (app)-based executive functioning tasks and an associative memory task 3 times over 2 weeks. Observational research participants were enrolled through 18 centers of a North American FTLD research consortium (ALLFTD) and were asked to complete the tests remotely using their own smartphones. Of 1163 eligible individuals (enrolled in parent studies), 360 were enrolled in the present study; 364 refused and 439 were excluded. Participants were divided into discovery (n = 258) and validation (n = 102) cohorts. Among 329 participants with data available on disease stage, 195 were asymptomatic or had preclinical FTLD (59.3%), 66 had prodromal FTLD (20.1%), and 68 had symptomatic FTLD (20.7%) with a range of clinical syndromes.

Exposure: Participants completed standard in-clinic measures and remotely administered ALLFTD mobile app (app) smartphone tests.

Main outcomes and measures: Internal consistency, test-retest reliability, association of smartphone tests with criterion standard clinical measures, and diagnostic accuracy.

Results: In the 360 participants (mean [SD] age, 54.0 [15.4] years; 209 [58.1%] women), smartphone tests showed moderate-to-excellent reliability (intraclass correlation coefficients, 0.77-0.95). Validity was supported by association of smartphones tests with disease severity (r range, 0.38-0.59), criterion-standard neuropsychological tests (r range, 0.40-0.66), and brain volume (standardized β range, 0.34-0.50). Smartphone tests accurately differentiated individuals with dementia from controls (area under the curve [AUC], 0.93 [95% CI, 0.90-0.96]) and were more sensitive to early symptoms (AUC, 0.82 [95% CI, 0.76-0.88]) than the Montreal Cognitive Assessment (AUC, 0.68 [95% CI, 0.59-0.78]) (z of comparison, -2.49 [95% CI, -0.19 to -0.02]; P = .01). Reliability and validity findings were highly similar in the discovery and validation cohorts. Preclinical participants who carried pathogenic variants performed significantly worse than noncarrier family controls on 3 app tasks (eg, 2-back β = -0.49 [95% CI, -0.72 to -0.25]; P < .001) but not a composite of traditional neuropsychological measures (β = -0.14 [95% CI, -0.42 to 0.14]; P = .32).

Conclusions and relevance: The findings of this cohort study suggest that smartphones could offer a feasible, reliable, valid, and scalable solution for remote evaluations of FTLD and may improve early detection. Smartphone assessments should be considered as a complementary approach to traditional in-person trial designs. Future research should validate these results in diverse populations and evaluate the utility of these tests for longitudinal monitoring.

Figure 1.. Screenshots of Smartphone Cognitive Tests and Testing Schedule

Screenshots of the smartphone cognitive tasks developed by Datacubed Health and included in the ALLFTD Mobile App. Details about the task design and instructions are included in the eMethods in Supplement 1. A, Flanker (Ducks in a Pond) is a task of cognitive control requiring participants to select the direction of the center duck. B, Go/no-go (Go Sushi Go!) requires participants to quickly tap on pieces of sushi (go) but not to tap when they see a fish skeleton (no-go). C, Card sort (Card Shuffle) is a task of cognitive flexibility requiring participants to learn rules that change during the task. D, The adaptative, associative memory task (Humi’s Bistro) requires participants to learn the food orders of several restaurant tables. E, Stroop (Color Clash) is a cognitive inhibition paradigm requiring participants to inhibit their tendency to read words and instead respond based on the color of the word. F, The 2-back task (Animal Parade) requires participants to determine whether animals on a parade float match the animals they saw 2 stimuli previously. G, Participants are asked to complete 3 testing sessions over 2 weeks. Shown in dark blue, they have 3 days to complete each testing session with a washout day between sessions on which no tests are available. Session 2 always begins on day 5 and session 3 on day 9. Screenshots are provided with permission from Datacubed Health.

Figure 2.. Reliability of Smartphone Cognitive Tests in a Mixed Sample of Adults Without Functional Impairment and Participants With Frontotemporal Lobar Degeneration

Forest plots present internal consistency and test-retest reliability results in the discovery and validation cohorts, as well as an estimate in a combined sample of discovery and validation participants. ICC indicates interclass correlation coefficient.

Figure 3.. Association of Smartphone Cognitive Tests With Criterion Standards and Detection of Deficits in Early Frontotemporal Lobar Degeneration (FTLD)

A and B, Correlation matrices display associations of in-clinic criterion standard measures and ALLFTD mobile App (mApp) test scores in discovery and validation cohorts. Below the horizontal dashed lines, the associations among app tests and between app tests and demographic characteristics convergent clinical measures, divergent cognitive tests, and neuroimaging regions of interest can be viewed. Most app tests show strong correlations with each other and with age, convergent clinical measures, and brain volume. The measures show weaker correlations with divergent measures of visuospatial (Benson Figure Copy) and language (Multilingual Naming Test [MINT]) abilities. The strength of convergent correlations between app measures and outcomes is similar to the correlations between criterion standard neuropsychological scores and these outcomes, which can be viewed by looking across the rows above the horizontal black line. C and D, In the discovery and validation cohorts, receiver operating characteristics curves were calculated to determine how well a composite of app tests, the Uniform Data Set, version 3.0, Executive Functioning Composite (UDS3-EF), and the Montreal Cognitive Assessment (MoCA) discriminate individuals without symptoms (Clinical Dementia Rating Scale plus National Alzheimer’s Coordinating Center FTLD module sum of boxes [CDR plus NACC-FTLD-SB] score = 0) from individuals with the mildest symptoms of FTLD (CDR plus NACC-FTLD-SB score = 0.5). AUC indicates area under the curve; CVLT, California Verbal Learning Test.