Electrophysiological differences and similarities in audiovisual speech processing in CI users with unilateral and bilateral hearing loss

Abstract

Hearing with a cochlear implant (CI) is limited compared to natural hearing. Although CI users may develop compensatory strategies, it is currently unknown whether these extend from auditory to visual functions, and whether compensatory strategies vary between different CI user groups. To better understand the experience-dependent contributions to multisensory plasticity in audiovisual speech perception, the current event-related potential (ERP) study presented syllables in auditory, visual, and audiovisual conditions to CI users with unilateral or bilateral hearing loss, as well as to normal-hearing (NH) controls. Behavioural results revealed shorter audiovisual response times compared to unisensory conditions for all groups. Multisensory integration was confirmed by electrical neuroimaging, including topographic and ERP source analysis, showing a visual modulation of the auditory-cortex response at N1 and P2 latency. However, CI users with bilateral hearing loss showed a distinct pattern of N1 topography, indicating a stronger visual impact on auditory speech processing compared to CI users with unilateral hearing loss and NH listeners. Furthermore, both CI user groups showed a delayed auditory-cortex activation and an additional recruitment of the visual cortex, and a better lip-reading ability compared to NH listeners. In sum, these results extend previous findings by showing distinct multisensory processes not only between NH listeners and CI users in general, but even between CI users with unilateral and bilateral hearing loss. However, the comparably enhanced lip-reading ability and visual-cortex activation in both CI user groups suggest that these visual improvements are evident regardless of the hearing status of the contralateral ear.

Keywords: Audiovisual speech perception; Bilateral hearing loss; Cochlear implant; Cortical plasticity; Event-related potential; Multisensory integration; Single-sided-deafness.

Graphical abstract

Fig. 1

Behavioural results. A) Overview of the paradigm (adapted from Layer et al., 2022). B) Mean response times for auditory (red), visual (green) and audiovisual (blue) syllables averaged over all groups, demonstrating that audiovisual syllables had shorter response times than auditory-only and visual-only RTs. C) Mean hit rates for auditory (red), visual (green) and audiovisual (blue) syllables averaged across all groups, with no differences between the three conditions. D) Cumulative distribution functions for CI-CHD, CI-SSD and NH. The race model is violated for all three groups because they show that the probability of faster response times is higher for audiovisual stimuli (blue line) than for those estimated by the race model (cyan line). Significant differences are indicated (* $p<0.05$, ** $p<0.01$, *** $p<0.001$). (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

Fig. 2

ERP results on the sensor level. A) GFP of conditions A and AV-V for CI-CHD users (red), CI-SSD users (green) and NH listeners (blue), including standard error. It is important to note that the GFP only provides positive values because it represents the standard deviation across all electrodes separately for each time point. The ERP topographies at the GFP peaks (N1(A) = CI-CHD: 147 ms, CI-SSD: 136 ms, NH: 118 ms; N1(AV-V) = CI-CHD: 137 ms, CI-SSD: 135 ms, NH: 118 ms; P2(A) = CI-CHD: 305 ms, CI-SSD: 288 ms, NH: 256 ms; P2(AV-V) = CI-CHD: 284 ms, CI-SSD: 307 ms, NH: 245 ms) are given separately for each group (displayed on the right). The grey-shaded areas represent the N1 and P2 time windows for detecting peak and latency. The grey bars below represent the time window in which significant GMDs between the three groups were observed. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

Fig. 3

Results from the hierarchical clustering and the single-subject fitting. A) Cumulative map frequency of the N1 maps: the CI-CHD users, but not the NH listeners or CI-SSD users, show a condition effect, with more frequent N1 map presence for AV-V compared to A. The corresponding map topographies are displayed on the right side, with Map B being referred to as the N1 topography. B) Cumulative map frequency of the P2 maps: there is a group effects (independent of the condition): NH listeners reveal a more frequent presence of a P2 topography (Map C) compared to CI-SSD users, and CI-SSD users show a more frequent presence of a P2-like topography (Map E) compared to NH listeners. CI-CHD users show a more frequent presence of Map D compared to NH listeners. Additionally, there is a condition effect (independent of the group): The presence of the P2-like topography (Map E) is enhanced for AV-V compared to A. This suggests a visual modulation of auditory speech processing at P2 latency in all groups. Significant differences are indicated (* $p<0.05$, ** $p<0.01$, *** $p<0.001$).

Fig. 4

ERP results on the source level. A) N1 and P2 source activity for CI users (red), CI-SSD users (green) and NH listeners (blue) separately for each ROI and each hemisphere with standard error (standard error shading was capped at zero). The source activity is displayed as absolute values with arbitrary units based on the normalisation within Brainstorm's dSPM algorithm. The grey shaded areas mark the N1 (light grey) and the P2 (dark grey) time windows. The boxes depict the location of the defined ROIs, with auditory ROIs in blue and visual ROIs in yellow. B) Group effect of the N1 peak mean in the visual cortex: both CI-CHD and CI-SSD users show more activity in the visual cortex compared to NH listeners, regardless of condition. C) Condition effect of the N1 peak mean in the auditory cortex: there is a significantly reduced auditory-cortex activation for AV-V compared to A, indicating multisensory interactions in all groups. D) N1 latency effect in the auditory cortex: Both CI and CI-SSD users show a prolonged N1 latency compared to NH listeners in the auditory cortex, regardless of the condition. This suggests a delayed auditory-cortex activation in CI users, independent of the hearing threshold in the contralateral ear. E) P2 condition effect in the visual cortex: there is a significantly reduced visual-cortex activation for AV-V compared to A, pointing towards multisensory interactions in all groups. Significant differences are indicated (* $p<0.05$, ** $p<0.01$, *** $p<0.001$). (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)