Intermodal event files: integrating features across vision, audition, taction, and action

Abstract

Understanding how the human brain integrates features of perceived events calls for the examination of binding processes within and across different modalities and domains. Recent studies of feature-repetition effects have demonstrated interactions between shape, color, and location in the visual modality and between pitch, loudness, and location in the auditory modality: repeating one feature is beneficial if other features are also repeated, but detrimental if not. These partial-repetition costs suggest that co-occurring features are spontaneously bound into temporary event files. Here, we investigated whether these observations can be extended to features from different sensory modalities, combining visual and auditory features in Experiment 1 and auditory and tactile features in Experiment 2. The same types of interactions, as for unimodal feature combinations, were obtained including interactions between stimulus and response features. However, the size of the interactions varied with the particular combination of features, suggesting that the salience of features and the temporal overlap between feature-code activations plays a mediating role.

Fig. 1

Sequence of events in Experiment 1. A visual response cue signaled a left or right mouse button click (R1) that was to be delayed until presentation of an audiovisual stimulus S1 (S1 is used as a detection signal for R1). The audiovisual stimulus S2 appeared 450 ms after R1. S2 signaled R2, a speeded left or right mouse button click according to the instructed mapping and task

Fig. 2

Reaction times of R2 in Experiment 1 as a function of repetition versus alternation of the stimuli (S1–S2) of visual feature color and auditory feature pitch, regardless of the response

Fig. 3

Reaction times of R2 in Experiment 1 for repetition versus alternation of the stimuli in the auditory feature pitch and the visual feature color, as a function of response repetition (vs. alternation) and task

Fig. 4

Reaction times of R2 in Experiment 2 as a function of repetition versus alternation of the stimuli (S1–S2) of tactile feature vibration and auditory feature pitch, and task

Fig. 5

Reaction times of R2 in Experiment 2 for repetition versus alternation of the stimuli in the auditory feature pitch and the tactile feature vibration, as a function of response repetition (vs. alternation) and task