Visual adaptation enhances action sound discrimination

Abstract

Prolonged exposure, or adaptation, to a stimulus in 1 modality can bias, but also enhance, perception of a subsequent stimulus presented within the same modality. However, recent research has also found that adaptation in 1 modality can bias perception in another modality. Here, we show a novel crossmodal adaptation effect, where adaptation to a visual stimulus enhances subsequent auditory perception. We found that when compared to no adaptation, prior adaptation to visual, auditory, or audiovisual hand actions enhanced discrimination between 2 subsequently presented hand action sounds. Discrimination was most enhanced when the visual action "matched" the auditory action. In addition, prior adaptation to a visual, auditory, or audiovisual action caused subsequent ambiguous action sounds to be perceived as less like the adaptor. In contrast, these crossmodal action aftereffects were not generated by adaptation to the names of actions. Enhanced crossmodal discrimination and crossmodal perceptual aftereffects may result from separate mechanisms operating in audiovisual action sensitive neurons within perceptual systems. Adaptation-induced crossmodal enhancements cannot be explained by postperceptual responses or decisions. More generally, these results together indicate that adaptation is a ubiquitous mechanism for optimizing perceptual processing of multisensory stimuli.

Keywords: Action; Adaptation; Audition; Crossmodal; Discrimination; Multimodal; Perception; Vision.

Fig. 1

Stimuli and experimental procedure for Experiment 1. (a) Waveforms show audible component of knock and slap action sounds. X-axis shows stimulus duration of 680 ms and sound onset at 200 ms. Images illustrate grayscale versions of individual frames (left to right: 2, 4, 7) from each visual action. (b) Schematic description of the experimental procedure in Experiment 1 for the V adapting condition. Following preadaptation on the first trial and top-up adaptation on subsequent trials to visual knocks, two slightly different knock action sounds (the standard and the comparison) were presented sequentially. Participants were required to indicate which sound appeared most like a knock

Fig. 2

Adaptation improves action sound discrimination. Left panel: psychometric functions fitted to the data of an example individual under no adapt (black), AVc (dark gray), V (mid-gray) and A (light gray) conditions. The circles illustrate data points, where circle magnitude corresponds to the number of trials for that data point. Cumulative Gaussian functions are fitted to the data, the slopes of which are used to derive JNDs; these show that adaptation increases the ability of this individual to discriminate the knock action sounds. Right panel: average JNDs across all participants tested. Error bars denote ±SEM. Asterisks denote a significant difference between conditions based upon planned contrasts. ***p < .005. *p < .05

Fig. 3

Action type influences action sound discrimination. Left panel: psychometric functions fitted to the data for an example individual under no adapt (black), adapt Vknock (light gray) and adapt Vslap (dark gray) conditions. The circles illustrate data points, where circle magnitude corresponds to the number of trials for that data point. Cumulative Gaussian functions are fitted to the data points, the slopes of which are used to derive JNDs and show that adaptation increases the ability of this individual to discriminate the knock action sounds. Right panel: average JNDs across all participants tested. Error bars denote ±SEM. Asterisks denote a significant difference between the Vknock and Vslap conditions. *p < .05

Fig. 4

Auditory aftereffects generated by unimodal and multimodal adaptation. Ambiguous action sound perception following adaptation to knocks (dark bars) and slaps (white bars). Adapting stimuli were presented as auditory stimuli alone (A), as congruent audiovisual stimuli (AVc) where the visual stimulus matched the sound, as incongruent audiovisual stimuli (AVi) where the visual action was incongruent to the sound, and as visual stimuli alone (V). Error bars denote ±SEM. Asterisks denote a significant difference between action conditions for each adaptor modality. ****p < .001. *p < .05

Fig. 5

Adaptation to visual actions, but not action words, biases action sound perception. Ambiguous action sound perception following adaptation to knocks (dark bars) and slaps (white bars). Adapting stimuli were presented as congruent audiovisual stimuli (AVc), as visual stimuli alone (V), or as orthographically presented action words (O). Error bars denote ±SEM. Asterisks denote a significant difference between action conditions for each adaptor modality. ****p < .001. ***p < .005