The integration of continuous audio and visual speech in a cocktail-party environment depends on attention

Abstract

In noisy environments, our ability to understand speech benefits greatly from seeing the speaker's face. This is attributed to the brain's ability to integrate audio and visual information, a process known as multisensory integration. In addition, selective attention plays an enormous role in what we understand, the so-called cocktail-party phenomenon. But how attention and multisensory integration interact remains incompletely understood, particularly in the case of natural, continuous speech. Here, we addressed this issue by analyzing EEG data recorded from participants who undertook a multisensory cocktail-party task using natural speech. To assess multisensory integration, we modeled the EEG responses to the speech in two ways. The first assumed that audiovisual speech processing is simply a linear combination of audio speech processing and visual speech processing (i.e., an A + V model), while the second allows for the possibility of audiovisual interactions (i.e., an AV model). Applying these models to the data revealed that EEG responses to attended audiovisual speech were better explained by an AV model, providing evidence for multisensory integration. In contrast, unattended audiovisual speech responses were best captured using an A + V model, suggesting that multisensory integration is suppressed for unattended speech. Follow up analyses revealed some limited evidence for early multisensory integration of unattended AV speech, with no integration occurring at later levels of processing. We take these findings as evidence that the integration of natural audio and visual speech occurs at multiple levels of processing in the brain, each of which can be differentially affected by attention.

Keywords: Cocktail party; Hierarchical processing; Multisensory integration; Speech.

Fig. 1.. Experimental procedure and analysis approach.

Silhouette images are used to protect identity of the speakers. A. Participants were presented with a male speaker in −9 dB noise in audiovisual (AV), audio-only (A) and video-only (V) format. Envelope reconstruction models AV and (A + V) decoders were derived from their EEG recordings. The channel weightings for each decoder averaged across time-lags from 0 to 500 ms and across all participants (N = 21) are shown on the right. B. Part of this figure is replicated from O’Sullivan et al., 2019. A separate set of participants performed a multisensory cocktail-party task where they were presented with an audiovisual speaker (A1V1) directly in front of them and an audio only speaker (A1) at 30° to their right (over headphones). They had to either attend or ignore A1V1. The AV and A + V decoders obtained from the single-speaker paradigm were applied to reconstruct the envelope of the audiovisual speaker (A1) when he was attended vs unattended. Multisensory integration in both conditions was quantified as the difference between reconstruction accuracy (Pearson’s r between the actual and reconstructed envelope of A1) using the AV decoder (r_AV) and A + V decoder (r_{A + V} ).

Fig. 2.. Robust multisensory integration when AV speech is attended vs unattended.

A. Both attended (left) and unattended (right) audiovisual speech envelope could be reconstructed significantly better than chance (determined by permutation test). B. Grand-average (N = 17) normalized multisensory gain was positive for attended, and negative for unattended audiovisual speech. Error bars indicate standard error of the mean C. Normalized multisensory gain at single-participant level. The sharp dissociation across attention (more positive multisensory gain for attended, than for unattended; trial-averaged) was visible for all but participant 4. D. Topographical distribution of EEG prediction accuracies (obtained from forward modeling). The black markers indicate channels where the multisensory integration effect [AV – (A + V)] was significant across participants (p < 0.05, two-sided cluster-based permutation test). *** p < 0.001, ** p < 0.01, * p < 0.05.

Fig. 3.. Spatiotemporal analysis reveals more complex dissociation of multisensory interaction across attention conditions.

A. Single-lag reconstruction accuracies obtained using AV, A and V decoders at every lag between −500 and 500 msec using the single-speaker in noise data. B. Single-lag reconstruction accuracies obtained using AV and (A + V) decoders at every lag between −500 and 500 msec using the cocktail party data. The green and the purple dots indicate time lags when the reconstruction accuracies of the speech envelope are significantly above chance using the AV and (A + V) decoders respectively (p <0.01, permutation tests). The gray rectangle running along the bottom of each panel indicates time lags when the statistical comparison between performances of AV and (A + V) decoders were made. The black markers in each panel indicate time lags when the multisensory integration effect [AV-(A + V)] was significant across participants (p < 0.05, paired t-test, FDR corrected). C. Topographic distributions of EEG prediction accuracies at specific time lags.