Integrating Biomarkers From Virtual Reality and Magnetic Resonance Imaging for the Early Detection of Mild Cognitive Impairment Using a Multimodal Learning Approach: Validation Study

Abstract

Background: Early detection of mild cognitive impairment (MCI), a transitional stage between normal aging and Alzheimer disease, is crucial for preventing the progression of dementia. Virtual reality (VR) biomarkers have proven to be effective in capturing behaviors associated with subtle deficits in instrumental activities of daily living, such as challenges in using a food-ordering kiosk, for early detection of MCI. On the other hand, magnetic resonance imaging (MRI) biomarkers have demonstrated their efficacy in quantifying observable structural brain changes that can aid in early MCI detection. Nevertheless, the relationship between VR-derived and MRI biomarkers remains an open question. In this context, we explored the integration of VR-derived and MRI biomarkers to enhance early MCI detection through a multimodal learning approach.

Objective: We aimed to evaluate and compare the efficacy of VR-derived and MRI biomarkers in the classification of MCI while also examining the strengths and weaknesses of each approach. Furthermore, we focused on improving early MCI detection by leveraging multimodal learning to integrate VR-derived and MRI biomarkers.

Methods: The study encompassed a total of 54 participants, comprising 22 (41%) healthy controls and 32 (59%) patients with MCI. Participants completed a virtual kiosk test to collect 4 VR-derived biomarkers (hand movement speed, scanpath length, time to completion, and the number of errors), and T₁-weighted MRI scans were performed to collect 22 MRI biomarkers from both hemispheres. Analyses of covariance were used to compare these biomarkers between healthy controls and patients with MCI, with age considered as a covariate. Subsequently, the biomarkers that exhibited significant differences between the 2 groups were used to train and validate a multimodal learning model aimed at early screening for patients with MCI among healthy controls.

Results: The support vector machine (SVM) using only VR-derived biomarkers achieved a sensitivity of 87.5% and specificity of 90%, whereas the MRI biomarkers showed a sensitivity of 90.9% and specificity of 71.4%. Moreover, a correlation analysis revealed a significant association between MRI-observed brain atrophy and impaired performance in instrumental activities of daily living in the VR environment. Notably, the integration of both VR-derived and MRI biomarkers into a multimodal SVM model yielded superior results compared to unimodal SVM models, achieving higher accuracy (94.4%), sensitivity (100%), specificity (90.9%), precision (87.5%), and F₁-score (93.3%).

Conclusions: The results indicate that VR-derived biomarkers, characterized by their high specificity, can be valuable as a robust, early screening tool for MCI in a broader older adult population. On the other hand, MRI biomarkers, known for their high sensitivity, excel at confirming the presence of MCI. Moreover, the multimodal learning approach introduced in our study provides valuable insights into the improvement of early MCI detection by integrating a diverse set of biomarkers.

Keywords: MRI; VR; early detection; eye movement; hand movement; magnetic resonance imaging; mild cognitive impairment; multimodal learning; virtual reality.

Figure 1

Experimental setup for the virtual kiosk test. The virtual kiosk test is operated via a laptop. Participants sit and wear a head-mounted display and interact with the virtual environment using a hand controller. Their hand movements, eye movements, and performance data are tracked via base stations.

Figure 2

The 6 sequential steps of the virtual kiosk test. In step 1, participants selected a place to eat. In step 2, they chose a burger item. Step 3 involved selecting a side item, and in step 4, participants chose a drink item. Step 5 required them to select a payment method, and finally, in step 6, they entered the payment password.

Figure 3

Extraction of 4 virtual reality (VR)–derived biomarkers from behavioral data in the virtual kiosk test. Hand movement speed is calculated using the hand movement data collected from the virtual kiosk test. Scanpath length is derived from the eye movement data. The time to completion and the number of errors are calculated based on the performance data.

Figure 4

Extraction of 22 magnetic resonance imaging (MRI) biomarkers from both hemispheres of the brain using the Split-Attention U-Net architecture. Following multilabel segmentation of the region of interest in the brain, each brain region’s volume is quantified as an MRI biomarker. Each hemisphere has 11 biomarkers including the cerebral white matter; cerebral gray matter; ventricles; amygdala; hippocampus; entorhinal cortex; parahippocampal gyrus; fusiform gyrus; and superior, middle, and inferior temporal gyrus.

Figure 5

Significant correlations between 4 virtual reality–derived biomarkers (hand movement speed, scanpath length, time to completion, and the number of errors) and the magnetic resonance imaging biomarker, specifically, the left hippocampus. HC: healthy controls; MCI: mild cognitive impairment.

Figure 6

Comparison of receiver operating characteristic curves and the area under the receiver operating characteristic curve (AUC). The best classification performance was obtained when the support vector machine was trained using a combination of virtual reality (VR)–derived biomarkers (hand movement speed, scanpath length, and the number of errors) and magnetic resonance imaging (MRI) biomarkers (left entorhinal cortex and left hippocampus). The gold-standard Seoul Neuropsychological Screening Battery–Core (SNSB-C; Rey Complex Figure Test and Seoul Verbal Learning Test–Elderly’s Version–Delayed Recall) was omitted from this comparison.

Figure 7

Comparison of hand movements, eye movements, and T1-weighted magnetic resonance imaging (MRI) between healthy controls and patients with mild cognitive impairment (MCI). (A) 3D coordinates of hand movements (depicted in blue). (B) Participant focus points indicated by dots, with red, blue, and purple representing the start, middle, and end of gaze, respectively—dot size corresponds to fixation duration. (C) Patients with MCI exhibiting statistically significant atrophy in the amygdala, hippocampus, and entorhinal cortex compared to healthy controls.