The effect of synesthetic associations between the visual and auditory modalities on the Colavita effect

Abstract

The Colavita effect refers to the phenomenon that when confronted with an audiovisual stimulus, observers report more often to have perceived the visual than the auditory component. The Colavita effect depends on low-level stimulus factors such as spatial and temporal proximity between the unimodal signals. Here, we examined whether the Colavita effect is modulated by synesthetic congruency between visual size and auditory pitch. If the Colavita effect depends on synesthetic congruency, we expect a larger Colavita effect for synesthetically congruent size/pitch (large visual stimulus/low-pitched tone; small visual stimulus/high-pitched tone) than synesthetically incongruent (large visual stimulus/high-pitched tone; small visual stimulus/low-pitched tone) combinations. Participants had to identify stimulus type (visual, auditory or audiovisual). The study replicated the Colavita effect because participants reported more often the visual than auditory component of the audiovisual stimuli. Synesthetic congruency had, however, no effect on the magnitude of the Colavita effect. EEG recordings to congruent and incongruent audiovisual pairings showed a late frontal congruency effect at 400-550 ms and an occipitoparietal effect at 690-800 ms with neural sources in the anterior cingulate and premotor cortex for the 400- to 550-ms window and premotor cortex, inferior parietal lobule and the posterior middle temporal gyrus for the 690- to 800-ms window. The electrophysiological data show that synesthetic congruency was probably detected in a processing stage subsequent to the Colavita effect. We conclude that-in a modality detection task-the Colavita effect can be modulated by low-level structural factors but not by higher-order associations between auditory and visual inputs.

Keywords: Audiovisual integration; Colavita effect; Event-related potentials; Synesthetic congruency.

Fig. 1

a The mean error rates for the unimodal and audiovisual stimuli. The unimodal conditions comprised auditory high and low pitch, visual small and large stimulus. The audiovisual conditions were either congruent (visual small/auditory high; visual large/auditory low) or incongruent (visual small/auditory low; visual large/auditory high). The erroneous responses are denoted by response (Resp) A (auditory), V (visual) and AV (audiovisual). b RTs for the unimodal and congruent and incongruent audiovisual stimuli. Error bars ± SEM. c Cumulative RT distribution functions (CDF) for the A, V, congruent (AVC), incongruent (AVI) conditions and the race model predictions (sum of the A and V CDFs: A + V)

Fig. 2

a Point-wise running t tests of the AV difference wave (incongruent–congruent) tested against prestimulus baseline activity in a 1- to 800-ms window at fronto-polar (FP), frontal (F), fronto-central (FC), central (C), centro-parietal (CP), parietal (P) and occipital (O) regions. The plot is highlighted only if at least 12 consecutive points were significant. b Comparison between congruent (visual small/auditory high; visual large/auditory low) and incongruent (visual small/auditory low; visual large/auditory high) grand averaged ERPs at electrodes Fz and POz. The scalp distribution of the AV difference wave (incongruent–congruent) plotted in 400–550 ms, 690–730 ms and 760–800 ms windows

Fig. 3

Statistical comparison of LAURA source estimation between the congruent and incongruent audiovisual presentations in 400–550 ms, 690–730 ms and 760–800 ms windows