A virtual reality-based system integrated with fmri to study neural mechanisms of action observation-execution: a proof of concept study

Abstract

Purpose: Emerging evidence shows that interactive virtual environments (VEs) may be a promising tool for studying sensorimotor processes and for rehabilitation. However, the potential of VEs to recruit action observation-execution neural networks is largely unknown. For the first time, a functional MRI-compatible virtual reality system (VR) has been developed to provide a window into studying brain-behavior interactions. This system is capable of measuring the complex span of hand-finger movements and simultaneously streaming this kinematic data to control the motion of representations of human hands in virtual reality.

Methods: In a blocked fMRI design, thirteen healthy subjects observed, with the intent to imitate (OTI), finger sequences performed by the virtual hand avatar seen in 1st person perspective and animated by pre-recorded kinematic data. Following this, subjects imitated the observed sequence while viewing the virtual hand avatar animated by their own movement in real-time. These blocks were interleaved with rest periods during which subjects viewed static virtual hand avatars and control trials in which the avatars were replaced with moving non-anthropomorphic objects.

Results: We show three main findings. First, both observation with intent to imitate and imitation with real-time virtual avatar feedback, were associated with activation in a distributed frontoparietal network typically recruited for observation and execution of real-world actions. Second, we noted a time-variant increase in activation in the left insular cortex for observation with intent to imitate actions performed by the virtual avatar. Third, imitation with virtual avatar feedback (relative to the control condition) was associated with a localized recruitment of the angular gyrus, precuneus, and extrastriate body area, regions which are (along with insular cortex) associated with the sense of agency.

Conclusions: Our data suggest that the virtual hand avatars may have served as disembodied training tools in the observation condition and as embodied "extensions" of the subject's own body (pseudo-tools) in the imitation. These data advance our understanding of the brain-behavior interactions when performing actions in VE and have implications in the development of observation- and imitation-based VR rehabilitation paradigms.

Fig. 1

A sample of our currently developed virtual environments. A. Dining Table Scene, B. Piano trainer, C. The virtual environment used in the current paradigm. We extracted the essential component common to all of our virtual environments, the virtual hands, over a plane background. Below them is a picture of a subject’s hand wearing a 5DT data glove that actuated motion of the virtual hand models.

Fig. 2

Left Panes: Simple main effect of observation only in VR versus rest. In red are regions activated when subjects observed, with the intent to imitate (OTI), a virtual hand perform a natural pre-recorded finger sequence. In green are regions activated when subjects passively viewed a rotating ellipsoid (WATCH e, see Methods). Right Panes: Simple main effect of execution versus rest. In red are regions activated when subjects imitated the finger sequence (that they observed in the OTI condition) with real-time control of representations of their hands in VR (MOVE h). In green are regions activated when subjects performed the finger sequence while viewing rotating ellipsoids that were not controlled by the subject’s motion (MOVE e). All yellow colors depict regions where the activations in red and green overlapped. All activations are thresholded at p < 0.001 and extent of 10 voxels.

Fig. 3

Regions activated in the OTI > WATCH e (red) and MOVE h > MOVE e (green) contrasts.

Fig. 4

A. The top left panel shows on an SPM glass brain the only region, the insula, that showed a significant time-variant increase in activation during the OTI condition. The remaining three panels show bar plots of the beta values at three cortical locations: in the insula shown in the glass brain and two control sites that were recruited in the simple main effect contrast. Note that the time-variant increase is evident only in the insula and only in the OTI condition. The bar plots for the parietal site in the WATCH e condition are not shown since this site was not recruited in the simple main effect. B. The simultaneously recorded time-series data for the BOLD signal (top) (group mean ± 1SD) and the joint angles (bottom) (one representative subject) of the four fingers. Shaded vertical bars denote the condition epochs (wh, OTI; mh, MOVE h; we, WATCH e; me, MOVE e).