Processing of Visual Speech Cues in Speech-in-Noise Comprehension Depends on Working Memory Capacity and Enhances Neural Speech Tracking in Older Adults With Hearing Impairment

Abstract

Comprehending speech in noise (SiN) poses a challenge for older hearing-impaired listeners, requiring auditory and working memory resources. Visual speech cues provide additional sensory information supporting speech understanding, while the extent of such visual benefit is characterized by large variability, which might be accounted for by individual differences in working memory capacity (WMC). In the current study, we investigated behavioral and neurofunctional (i.e., neural speech tracking) correlates of auditory and audio-visual speech comprehension in babble noise and the associations with WMC. Healthy older adults with hearing impairment quantified by pure-tone hearing loss (threshold average: 31.85-57 dB, N = 67) listened to sentences in babble noise in audio-only, visual-only and audio-visual speech modality and performed a pattern matching and a comprehension task, while electroencephalography (EEG) was recorded. Behaviorally, no significant difference in task performance was observed across modalities. However, we did find a significant association between individual working memory capacity and task performance, suggesting a more complex interplay between audio-visual speech cues, working memory capacity and real-world listening tasks. Furthermore, we found that the visual speech presentation was accompanied by increased cortical tracking of the speech envelope, particularly in a right-hemispheric auditory topographical cluster. Post-hoc, we investigated the potential relationships between the behavioral performance and neural speech tracking but were not able to establish a significant association. Overall, our results show an increase in neurofunctional correlates of speech associated with congruent visual speech cues, specifically in a right auditory cluster, suggesting multisensory integration.

Keywords: EEG; age-related hearing loss; audio-visual speech; neural speech tracking; speech in noise; working memory capacity.

Figure 1.

Pure-tone audiometry. The audiogram depicts individual pure-tone thresholds at frequencies between 0.5 and 8 kHz. There is no systematic difference between hearing-aid users (HA; group average depicted in red) and non-hearing-aid users (nHA; group average depicted in blue). Stimulus presentation was limited to 100 dB, which explains the accumulation of data points at 8 kHz. Hearing-aid users were measured while having their devices on.

Figure 2.

Illustration of the stimulus presentation. The modalities differed in that audio-visual-babble (AVB) included a video sequence of the mouth and jaw movement whereas the audio-babble (AB) modality only contained auditory stimuli. Five sentences were presented, and after each, a pattern-matching task was applied. After every fifth sentence, a comprehension question was asked. There was a total of 30 items per modality.

Figure 3.

Speech in noise performance estimated by speech presentation modality, working memory capacity, age, speech tracking and hearing loss. A: No significant increase in pattern matching across the two speech presentation modalities was revealed by the model. B: No significant increase in comprehension performance across modalities was observed. Compared to pattern matching, for the comprehension task only after every 5^th trial, a comprehension question was asked, resulting in a six-level classification of performance (30 trials within each modality were presented in total). C: Working memory capacity was significantly positively associated with pattern matching, while age and hearing loss revealed the opposite relationship. Neural speech tracking was not significantly associated. D: Working memory capacity was significantly associated with comprehension performance, while neither age, neural speech tracking nor hearing loss explained variance in the comprehension performance. n.s. = not significant, *p < .05, **p < .01, ***p < .001.

Figure 4.

Topographic distribution and time course of neural speech tracking. A: Grand average cross-correlation functions of the right, the left and the frontal cluster. Significant time windows are marked as bars over the function. B: Topographic distribution and time course of the grand average cross-correlation in all three listening modalities from approximately 50 to 250 ms. C: Topographical distribution of the grand-average cross-correlation at the peaks at approximately 100 and 250 ms. Selected electrode clusters are marked with “•”. Warm colors denote positive- and cool colors negative correlations.

Figure 5.

Neural speech tracking across modalities and cluster. Neural speech tracking significantly differed across modalities. In the presence of congruent visual speech cues, cross-correlation coefficients increased, particularly in the right auditory-related cluster. n.s. = not significant, *p < .05, **p < .01, ***p < .001.