Audio-visual integration in noise: Influence of auditory and visual stimulus degradation on eye movements and perception of the McGurk effect

Abstract

Seeing a talker's face can aid audiovisual (AV) integration when speech is presented in noise. However, few studies have simultaneously manipulated auditory and visual degradation. We aimed to establish how degrading the auditory and visual signal affected AV integration. Where people look on the face in this context is also of interest; Buchan, Paré and Munhall (Brain Research, 1242, 162-171, 2008) found fixations on the mouth increased in the presence of auditory noise whilst Wilson, Alsius, Paré and Munhall (Journal of Speech, Language, and Hearing Research, 59(4), 601-615, 2016) found mouth fixations decreased with decreasing visual resolution. In Condition 1, participants listened to clear speech, and in Condition 2, participants listened to vocoded speech designed to simulate the information provided by a cochlear implant. Speech was presented in three levels of auditory noise and three levels of visual blurring. Adding noise to the auditory signal increased McGurk responses, while blurring the visual signal decreased McGurk responses. Participants fixated the mouth more on trials when the McGurk effect was perceived. Adding auditory noise led to people fixating the mouth more, while visual degradation led to people fixating the mouth less. Combined, the results suggest that modality preference and where people look during AV integration of incongruent syllables varies according to the quality of information available.

Keywords: Audio-visual integration; Eye movements; McGurk effect; Multisensory perception.

Fig. 1

Six separate areas of interest were used encompassing the hair/forehead, the left and right eye, the chin/cheeks, nose and the mouth

Fig. 2

Variability in perception of the McGurk effect across participants and stimuli. Participants have been ordered according to their average across the four stimuli. Averages for each stimulus across participants are also shown. Panel A shows data for Clear speech and Panel B shows data for Vocoded speech

Fig. 3

Percentage of dwell time in each area of interest according to Congruence and Stimulus. Panels A and B (Clear Condition) show data for Clear speech while Panels C and D (Vocoded Condition) show data for Vocoded speech

Fig. 4

Effects of auditory and visual noise on the percentage of McGurk responses and the percentage dwell time on the mouth. Panels A and B show data from Clear speech and Panels C and D show data from Vocoded speech

Fig. 5

Percentage of dwell time on mouth according to fixation-cross position and whether the McGurk effect was perceived. Panel A shows data from Clear speech and Panel B shows data from Vocoded speech. Error bars show 95% confidence intervals