Electrophysiological evidence for differences between fusion and combination illusions in audiovisual speech perception

Abstract

Incongruent audiovisual speech stimuli can lead to perceptual illusions such as fusions or combinations. Here, we investigated the underlying audiovisual integration process by measuring ERPs. We observed that visual speech-induced suppression of P2 amplitude (which is generally taken as a measure of audiovisual integration) for fusions was similar to suppression obtained with fully congruent stimuli, whereas P2 suppression for combinations was larger. We argue that these effects arise because the phonetic incongruency is solved differently for both types of stimuli.

Keywords: ERPs; P2 suppression; audiovisual speech integration; phonetic audiovisual (in)congruency.

Figure 1

Proportions of ‘b’, ‘g’, ‘d’ and ‘bg/gb’ responses per stimulus. Panel a depicts individual data (grey), averages (white) and standard errors of the mean (shaded areas). Significance of the pairwise comparisons for ‘correct’ vs. ‘incorrect’ responses per stimulus is indicated below the plots. Panel b shows the corresponding test statistics, P‐values and effect sizes (all significant after FDR correction). ‘Stim’ indicates stimulus type, and ‘CR’ indicates the correct response to a stimulus.

Figure 2

Auditory grand averages, AV – V difference waves and statistical comparisons. Panel a shows the waveforms at Cz for stimuli with auditory ‘b’ (left column) and auditory ‘g’ (right column). In panel b, time zero corresponds to sound onset, and grey horizontal bars represent significant differences between conditions. For each pairwise comparison, the ERPs from electrode Cz are overlaid.