How a lack of haptic feedback affects eye-hand coordination and embodiment in virtual reality

Abstract

Intuitively, we know that how we perceive and act in the world is profoundly affected when the lights go out. But what happens to visuomotor control when our sense of touch is taken away? Notably this happens in Virtual Reality (VR) and for prosthesis users. We test this question by combining VR and hand-, motion- and eye-tracking to give and deprive full haptic feedback to individuals with normal hand function during a validated object interaction task. Returning haptic feedback in VR generated eye-hand coordination more similar to real-world interactions. Interestingly, VR users and prosthesis users have both reported reduced feelings of embodiment towards their limbs. Therefore, we also quantified the sense of embodiment which increased with haptic feedback. We further reported a correlation between eye-hand coordination and an individual's sense of embodiment suggesting that the embodiment is experienced as the synchronized reception of sensory information and that eye-hand coordination measures are an objective way to quantify these experiences.

Keywords: Embodiment; Eye-hand coordination; Eye-tracking; Haptic feedback; Motion-tracking; Virtual reality.

Fig. 1

The virtual and real-world hands of participants in each of the two conditions during an interaction. In the No Haptic condition, participants saw (a) virtual hands, and wore (b) VR tracking gloves without a real pasta box. In the Haptic condition, participants saw (c) virtual hands, and wore (d) VR tracking gloves and felt a real pasta box.

Fig. 2

Participants wore a VR HMD and tracking gloves while doing the Pasta Box Task which includes the Reach, Grasp, Transport, and Release of a pasta box at 3 target locations. (a) Movement 1: Grasp from side cart (Start/End Target) and Release on Mid Shelf Target. (b) Movement 2: Grasp from Mid Shelf Target and Release on High Shelf Target. (c) Movement 3: Grasp on High Shelf Target and Release on Start/End Target. Informed consent was obtained from the participant (first author) captured in the images used for this figure.

Fig. 3

The segmentation of an object Movement into its Reach, Grasp, Transport, and Release phases is determined by the velocity of the object (orange trace), the velocity of the hand (grey trace), and distances to task relevant locations. Also shown are the approximate temporal locations defined by the terms Pick-up and Drop-off, and the Eye Arrival Latency (EAL) and Eye Leaving Latency (ELL) measures associated with each (adapted from (Lavoie et al., 2018)).

Fig. 4

The average (bar heights) of all participants’ (a) Absolute Duration(s), and (b) Relative Duration(%), of each phase of movement of the Pasta Box Task in the Haptic (filled bars) and No Haptic (unfilled bars) conditions. Individual participant data are shown with connected filled (Haptic) and unfilled (No Haptic) circles. In both (a) and (b) the Home phase is omitted as it is irrelevant to the object interaction.

Fig. 5

The average (bar heights) of all participants’ (a) Number of Fixations to Current(#), (b) % Fixation Time to Current(%), (c) Number of Fixations to Hand in Flight(#), and (d) % Fixation Time to Hand in Flight(%) of each phase of movement of the Pasta Box Task in the Haptic (filled bars) and No Haptic (unfilled bars) conditions. Individual participant data are shown with connected filled (Haptic) and unfilled (No Haptic) circles.

Fig. 6

The average (bar height) of all participants’ EAL Grasp(s), ELL PU(s), EAL DO(s), and ELL Release(s) of the Pasta Box Task in the Haptic (filled bars) and No Haptic (unfilled bars) conditions. Individual participant data are shown with connected filled (Haptic) and unfilled (No Haptic) circles. The positive (above 0) and negative (below 0) conventions for these measures are taken from the description provided in Fig. 3. Relative to the events listed on the x-axis, a positive value indicates how long the eye was at the event-location before the hand arrived and a negative value indicates how long the eye lingered at the event-location after the hand had left.

Fig. 7

The correlational analyses showing that % Fixation Time to Current during Transport significantly increases as three Embodiment statements increase and % Fixation Time to Hand in Flight during Transport significantly decreases as one Embodiment statement increases, between the Haptic and No Haptic conditions.