When ears drive hands: the influence of contact sound on reaching to grasp

Abstract

Background: Most research on the roles of auditory information and its interaction with vision has focused on perceptual performance. Little is known on the effects of sound cues on visually-guided hand movements.

Methodology/principal findings: We recorded the sound produced by the fingers upon contact as participants grasped stimulus objects which were covered with different materials. Then, in a further session the pre-recorded contact sounds were delivered to participants via headphones before or following the initiation of reach-to-grasp movements towards the stimulus objects. Reach-to-grasp movement kinematics were measured under the following conditions: (i) congruent, in which the presented contact sound and the contact sound elicited by the to-be-grasped stimulus corresponded; (ii) incongruent, in which the presented contact sound was different to that generated by the stimulus upon contact; (iii) control, in which a synthetic sound, not associated with a real event, was presented. Facilitation effects were found for congruent trials; interference effects were found for incongruent trials. In a second experiment, the upper and the lower parts of the stimulus were covered with different materials. The presented sound was always congruent with the material covering either the upper or the lower half of the stimulus. Participants consistently placed their fingers on the half of the stimulus that corresponded to the presented contact sound.

Conclusions/significance: Altogether these findings offer a substantial contribution to the current debate about the type of object representations elicited by auditory stimuli and on the multisensory nature of the sensorimotor transformations underlying action.

Figure 1. Experimental set-up.

This figure depicts the location at which the four infrared cameras were positioned together with the participants' start position and the stimulus location.

Figure 2. Kinematic results for Experiment 1.

Average movement duration (panel ‘a’), deceleration time (panel ‘b’) and closing time (panel ‘c’) for the congruent, control and incongruent sound conditions. Error bars represent standard error of means. Asterisks indicate significant differences.

Figure 3. Contact points variability for Experiment 1.

Graphical representation of contact points variability for the index finger and thumb for the control (panel ‘a’), the congruent (panel ‘b’) and the incongruent (panel ‘c’) sound condition. Single trials for a representative participant (n. 14) are depicted.

Figure 4. Graphical representation of contact points for the index finger and thumb in Experiment 2.

The considered measure was in which half of the stimulus object the index finger and thumb contacted the object. This measure was calculated on the basis of the end trajectories of the fingers. A representative example of sound congruent grasp for the wool/paper material combination is presented (participant n. 9). The dashed line indicates the stimulus midline.