Symbol Digit Modalities Test Variant in a Smartphone App for Persons With Multiple Sclerosis: Validation Study

Abstract

Background: The decline of cognitive processing speed (CPS) is a common dysfunction in persons with multiple sclerosis (MS). The Symbol Digit Modalities Test (SDMT) is widely used to formally quantify CPS. We implemented a variant of the SDMT in MS sherpa, a smartphone app for persons with MS.

Objective: The aim of this study was to investigate the construct validity and test-retest reliability of the MS sherpa smartphone variant of the SDMT (sSDMT).

Methods: We performed a validation study with 25 persons with relapsing-remitting MS and 79 healthy control (HC) subjects. In the HC group, 21 subjects were matched to the persons with MS with regard to age, gender, and education and they followed the same assessment schedule as the persons with MS (the "HC matched" group) and 58 subjects had a less intense assessment schedule to determine reference values (the "HC normative" group). Intraclass correlation coefficients (ICCs) were determined between the paper-and-pencil SDMT and its smartphone variant (sSDMT) on 2 occasions, 4 weeks apart. Other ICCs were determined for test-retest reliability, which were derived from 10 smartphone tests per study participant, with 3 days in between each test. Seven study participants with MS were interviewed regarding their experiences with the sSDMT.

Results: The SDMT scores were on average 12.06% higher than the sSDMT scores, with a standard deviation of 10.68%. An ICC of 0.838 was found for the construct validity of the sSDMT in the combined analysis of persons with MS and HC subjects. Average ICCs for test-retest reliability of the sSDMT for persons with MS, the HC matched group, and the HC normative group were 0.874, 0.857, and 0.867, respectively. The practice effect was significant between the first and the second test of the persons with MS and the HC matched group and trivial for all other test-retests. The interviewed study participants expressed a positive attitude toward the sSDMT, but they also discussed the importance of adapting a smartphone cognition test in accordance with the needs of the individual persons with MS.

Conclusions: The high correlation between sSDMT and the conventional SDMT scores indicates a very good construct validity. Similarly, high correlations underpin a very good test-retest reliability of the sSDMT. We conclude that the sSDMT has the potential to be used as a tool to monitor CPS in persons with MS, both in clinical studies and in clinical practice.

Keywords: cognition; mobile phone; processing speed; relapsing-remitting multiple sclerosis.

Figure 1

Overview of the study design and assessment scheme. SDMT: Symbol Digit Modalities Test; 2MWT: 2-minute walking test.

Figure 2

Distributions in the number of correct answers on the first test done with the sSDMT for the 3 groups in this study. The thin solid line shows the distribution for persons with multiple sclerosis, the dotted line shows the distribution for the healthy control matched group, and the dashed line shows the distribution for the healthy control normative group. The thick solid lines represent Gaussian fits to the distributions, of which the means (SD) are shown in the legend. sSDMT: smartphone variant of Symbol Digit Modalities Test.

Figure 3

Distribution of the differences between the number of correct answers on the sSDMT and the SDMT. The dashed line represents a normal distribution. sSDMT: smartphone variant of Symbol Digit Modalities Test; SDMT: Symbol Digit Modalities Test.

Figure 4

Bland-Altman plot of differences between the number of correct answers on the sSDMT and SDMT, expressed as percentages of the mean value (∆/mean) versus the mean of the 2 measurements (raw data in Multimedia Appendix 1). The dashed line shows the mean percentage difference, and the dotted lines show the 95% confidence interval. sSDMT: smartphone variant of Symbol Digit Modalities Test; SDMT: Symbol Digit Modalities Test.

Figure 5

Scatter plot showing the ICCs(3,1) and the correlation (Pearson r, upper left corner) between the number of correct answers on the SDMT (horizontal axis, “x”) and the sSDMT (vertical axis, “y”). A linear fit through the data points is visualized as a black solid line, for which the formula is also given in the upper left corner. ICC: intraclass correlation coefficient; sSDMT: smartphone variant of Symbol Digit Modalities Test; SDMT: Symbol Digit Modalities Test.

Figure 6

A: Distribution of the number of correct answers on the first sSDMT of 13 persons with multiple sclerosis who used a smartphone with a large screen size (dashed line) and 13 persons with multiple sclerosis who used a smartphone with a small screen size (dotted line). The thick solid lines represent Gaussian fits to the distributions. B: Distribution of the number of correct answers on the first sSDMT of 11 persons with multiple sclerosis who used a smartphone with an iPhone operating system (iOS) (dotted line) and 15 persons with multiple sclerosis who used a smartphone with an Android operating system (dashed line). The thick solid lines represent Gaussian fits to the distributions. In both panels, the dot-dashed line represents a Gaussian fit to the full distribution (both categories). MS: multiple sclerosis; sSDMT: smartphone variant of Symbol Digit Modalities Test.

Figure 7

A: Histogram showing the distribution of total number of tests done. B: Histogram showing the distribution of number of days in between tests, averaged over all tests of a study participant. C: Scatter plot that shows the relation between the number of tests done (horizontal axis) and the average number of days in between tests, averaged over all tests of a study participant (vertical axis) for all study participants. Filled circles correspond to participants that finished the study within 28 days, open circles correspond to those for which the study duration exceeded 28 days. The blue lines in panels A and B correspond to these filled circles, and the black lines in panels A and B correspond to the open circles.