Neural Decoding of the Speech Envelope: Effects of Intelligibility and Spectral Degradation

Abstract

During continuous speech perception, endogenous neural activity becomes time-locked to acoustic stimulus features, such as the speech amplitude envelope. This speech-brain coupling can be decoded using non-invasive brain imaging techniques, including electroencephalography (EEG). Neural decoding may provide clinical use as an objective measure of stimulus encoding by the brain-for example during cochlear implant listening, wherein the speech signal is severely spectrally degraded. Yet, interplay between acoustic and linguistic factors may lead to top-down modulation of perception, thereby complicating audiological applications. To address this ambiguity, we assess neural decoding of the speech envelope under spectral degradation with EEG in acoustically hearing listeners (n = 38; 18-35 years old) using vocoded speech. We dissociate sensory encoding from higher-order processing by employing intelligible (English) and non-intelligible (Dutch) stimuli, with auditory attention sustained using a repeated-phrase detection task. Subject-specific and group decoders were trained to reconstruct the speech envelope from held-out EEG data, with decoder significance determined via random permutation testing. Whereas speech envelope reconstruction did not vary by spectral resolution, intelligible speech was associated with better decoding accuracy in general. Results were similar across subject-specific and group analyses, with less consistent effects of spectral degradation in group decoding. Permutation tests revealed possible differences in decoder statistical significance by experimental condition. In general, while robust neural decoding was observed at the individual and group level, variability within participants would most likely prevent the clinical use of such a measure to differentiate levels of spectral degradation and intelligibility on an individual basis.

Keywords: cochlear implants; cortical tracking; electroencephalography; objective measures; speech perception.

Figure 1.

Comparisons of English and Dutch speech stimuli. Panel A: Spectral power analysis of the English and Dutch versions of the story. Panel B: Speech amplitude envelopes extracted from unprocessed, vocoded, and vocoded + blurring listening conditions. Panel C: Histograms depicting the distributions of mean inter-vowel intervals (calculated over a 2 s-duration window) in the English and Dutch versions of the story. Panel D: Histograms depicting the distributions of coefficient of variation of inter-vowel intervals (calculated over a 2 s-duration window) in the English and Dutch versions of the story.

Figure 2.

Panel A: Box plots and group mean with bootstrap 95% confidence intervals depicting self-reported ability to follow the story. Panel B: Box plots and group mean with bootstrap 95% confidence intervals depicting self-reported engagement with the story. Panel C: Dot-and-whisker plot depicting standardized estimates (regression coefficients) with 95% confidence intervals from the linear mixed effects model of mean median reaction times in the repeated-phrase target detection task. The model reference levels were English unprocessed. Panel D: Box plots and group mean with bootstrap 95% confidence intervals depicting mean median reaction time by listening condition. Significance values reflect the outcome of pairwise tests of estimated marginal means between levels of spectral degradation with Bonferroni correction.

Figure 3.

Subject-specific speech decoding accuracy. Panel A: Dot-and-whisker plot depicting standardized estimates (regression coefficients) with 95% confidence intervals from the linear mixed effects model of subject-specific speech decoding accuracy. The model reference levels were English unprocessed (trained and test), with red colors depicting negative estimates, and blue, positive estimates. Panel B: Box plots and group mean with bootstrap 95% confidence intervals depicting speech decoding accuracy by trained: spectral degradation and test: spectral degradation. Data are collapsed across language conditions. Panel C: Box plots and group mean with bootstrap 95% confidence intervals depicting speech decoding accuracy by trained: language and test: language. Data are collapsed across spectral degradation conditions. Significance values reflect the outcome of pairwise tests of estimated marginal means with Bonferroni correction.

Figure 4.

Same as Figure 3 but for group speech decoding accuracy.

Figure 5.

Histograms depicting the scaled difference in neural decoding accuracy when subtracting group-level from subject-specific values (panel A), or when subtracting non-matched from matched training and testing listening conditions (panels B and C). Scaling is performed by dividing the resulting differences by the subject-specific (panel A) or matched training and testing (panels B and C) decoding accuracy values.

Figure 6.

Bar plots indicating the significance level, determined via random permutation testing with n = 1000 iterations, at which neural decoding was decoded across listening conditions by subject-specific decoders (panel A) and group decoders (panel B). Trained and test data were from matching listening conditions.