Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Sep:2024:3030-3034.
doi: 10.21437/interspeech.2024-2288.

Analyzing Multimodal Features of Spontaneous Voice Assistant Commands for Mild Cognitive Impairment Detection

Nana Lin  1 , Youxiang Zhu  1 , Xiaohui Liang  1 , John A Batsis  2 , Caroline Summerour  2
Affiliations

Analyzing Multimodal Features of Spontaneous Voice Assistant Commands for Mild Cognitive Impairment Detection

Nana Lin et al. Interspeech. 2024 Sep.

Abstract

Mild cognitive impairment (MCI) is a major public health concern due to its high risk of progressing to dementia. This study investigates the potential of detecting MCI with spontaneous voice assistant (VA) commands from 35 older adults in a controlled setting. Specifically, a command-generation task is designed with pre-defined intents for participants to freely generate commands that are more associated with cognitive ability than read commands. We develop MCI classification and regression models with audio, textual, intent, and multimodal fusion features. We find the command-generation task outperforms the command-reading task with an average classification accuracy of 82%, achieved by leveraging multimodal fusion features. In addition, generated commands correlate more strongly with memory and attention subdomains than read commands. Our results confirm the effectiveness of the command-generation task and imply the promise of using longitudinal in-home commands for MCI detection.

Keywords: machine learning; mild cognitive impairment; multimodal model; speech analysis.

PubMed Disclaimer

Figures

Figure 1:
Figure 1:
MCI detection model using multimodal features extracted from voice assistant commands (L: Language, O: Orientation, V: Visuospatial, E: Executive function, M: Memory, A: Attention).
Figure 2:
Figure 2:
Box plot of command count comparison in command-reading (Task1) and command-generation (Task2) tasks.
See this image and copyright information in PMC

References

    1. World Health Organization. “Dementia”. 2023. [Online]. Available: https://www.who.int/news-room/fact-sheets/detail/dementia
    1. Breton A, Casey D, and Arnaoutoglou NA, “Cognitive tests for the detection of mild cognitive impairment (mci), the prodromal stage of dementia: Meta-analysis of diagnostic accuracy studies,” International journal of geriatric psychiatry, vol. 34, no. 2, pp. 233–242, 2019. - PubMed
    1. Rasmussen J and Langerman H, “Alzheimer’s disease–why we need early diagnosis,” Degenerative neurological and neuromuscular disease, pp. 123–130, 2019. - PMC - PubMed
    1. Fristed E, Skirrow C, Meszaros M, Lenain R, Meepegama U, Cappa S, Aarsland D, and Weston J, “Evaluation of a speech-based ai system for early detection of alzheimer’s disease remotely via smartphones,” medRxiv, pp. 2021–10, 2021.
    1. Vigo I, Coelho L, and Reis S, “Speech-and language-based classification of alzheimer’s disease: A systematic review,” Bioengineering, vol. 9, no. 1, p. 27, 2022. - PMC - PubMed

LinkOut - more resources