Neural signatures of visuo-motor integration during human-robot interactions

Abstract

Visuo-motor integration shapes our daily experience and underpins the sense of feeling in control over our actions. The last decade has seen a surge in robotically and virtually mediated interactions, whereby bodily actions ultimately result in an artificial movement. But despite the growing number of applications, the neurophysiological correlates of visuo-motor processing during human-machine interactions under dynamic conditions remain scarce. Here we address this issue by employing a bimanual robotic interface able to track voluntary hands movement, rendered in real-time into the motion of two virtual hands. We experimentally manipulated the visual feedback in the virtual reality with spatial and temporal conflicts and investigated their impact on (1) visuo-motor integration and (2) the subjective experience of being the author of one's action (i.e., sense of agency). Using somatosensory evoked responses measured with electroencephalography, we investigated neural differences occurring when the integration between motor commands and visual feedback is disrupted. Our results show that the right posterior parietal cortex encodes for differences between congruent and spatially-incongruent interactions. The experimental manipulations also induced a decrease in the sense of agency over the robotically-mediated actions. These findings offer solid neurophysiological grounds that can be used in the future to monitor integration mechanisms during movements and ultimately enhance subjective experience during human-machine interactions.

Keywords: bimanual movements; electroencephalography; robotics; sense of agency; somatosensory evoked potentials; source imaging; virtual reality; visuo-motor integration.

Figure 1

Experimental set-up and virtual reality environment. Participants sat at a table and hold the frame of the robotic device between the index and the thumb of each hand while performing continuous horizontal (left-right) hand movement for 15 s. (A) Participants' hands movements were translated in real time into the movement of two virtual hands, presented through a head-mounted display. To mimic the reality, the virtual hands were shown holding a block (B).

Figure 2

Somatosensory evoked potential to stimulation of the median nerve at the right wrist. Group-averaged SEP waveform across the four experimental conditions as recorded from one exemplar central electrode (C3), contralateral to the stimulation side. Responses exhibit the prototypical peaks, including the P45 and N60 components. The scalp topography is shown with the nasion upwards and left scalp leftwards and depicts the neural activity at the P45 peak, with electrode C3 encircled. The shaded time interval indicates the window of interest over which the neuroimaging analyses were performed.

Figure 3

Topographic clustering and “fitting” procedure. The topographic pattern analyses identified periods of stable topography across the collective 300 ms post-electrical stimulation onset. For one of these time periods (21–56 ms), two maps were identified from the group-averaged SEPs. One map most prominently accounted for the congruent condition [map (A), left], while the other for the incongruent one [map (B), right]. The reliability of this result, observed at the group-average level, was assessed at the single-subject level using a spatial correlation fitting procedure. This analysis revealed a significantly different duration for the two maps in the congruent (green) as compared to the incongruent (pink) condition (main effect of the congruency factor, whiskers plots).

Figure 4

Topographic consistency and global field power. Results of the topographic consistency tests (TCT, upper plot) show intervals of time in which the null hypothesis of observed topographies explained entirely by noise is rejected (p-values < 0.05). For displaying purposes, the y-axis is expressed in expressed in terms of 1-p values. The TCT across the four conditions show periods of stable topographies during two time-windows, respectively 20–75 and 150–220 ms post stimulus onset. The first-time window covers the onset and peak of global field power (GFP, lower plot), and occurs around the P45 evoked potential. The time interval highlighted indicates the interval of interest over which the neuroimaging analyses were performed.

Figure 5

Neural generators of somatosensory evoked response. Group-averaged LAURA source density estimations over the 21–56 ms period post-stimulus onset for each of the four experimental conditions. Results are displayed on an average MNI brain. All conditions exhibited neural generators located within the left somatosensory area, contralateral to the stimulation side (right wrist).

Figure 6

Statistical analyses of the source estimations. Group-averaged source estimations were calculated over the 21–56 ms post-stimulus interval for each experimental condition and submitted to a repeated measures 2 × 2 ANOVA performed in the brain space. Two clusters that exhibited a significant main effect of the factor congruency are shown in axial slices of the MNI template brain, in correspondence of the posterior parietal cortex (PPC, 37 nodes) and posterior cingulate cortex (PCC, 12 nodes). Only nodes meeting the p-values < 0.05 statistical threshold and the spatial criterion of at least 17 contiguous nodes were considered reliable. Whisker plots depict the current density values in the PPC cluster in the congruent (green) and incongruent (pink) conditions. Significance is denoted with ** for p < 0.01.

Figure 7

Sense of agency during human-robot interactions. Questionnaire results regarding the question on the sense of agency: the ANOVA revealed both a main effect of congruency and synchrony. Significance is denoted with ** for p < 0.01.

Figure 8

Movement trajectories as displayed in the virtual environment from one representative participant. During the task, participants were instructed to perform continuous horizontal left-right hand movements while interacting with the robotic interface. These movements were translated in real-time in the virtual reality environment, presented to the participant through a head-mounted display. To interfere with visuo-motor integration mechanisms, the visual feedback provided in the VR was either in accordance with the executed movement (congruent-synchronous condition, red trajectory) or experimentally manipulated. For this, a spatial mismatch translating the horizontal movement into a vertical one (incongruent conditions, black and blue trajectories) and a temporal delay (asynchronous conditions, green and blue trajectories) were employed in selected trials, resulting in a total of four experimental conditions. Trajectories are displayed for each condition during a single trial. Overall, the mean trajectory norm in the left-right direction was 10.6 cm, and the mean velocity 6.7 cm/s.