Objective Assessment of Listening Effort: Coregistration of Pupillometry and EEG

Abstract

Listening to speech in noise is effortful, particularly for people with hearing impairment. While it is known that effort is related to a complex interplay between bottom-up and top-down processes, the cognitive and neurophysiological mechanisms contributing to effortful listening remain unknown. Therefore, a reliable physiological measure to assess effort remains elusive. This study aimed to determine whether pupil dilation and alpha power change, two physiological measures suggested to index listening effort, assess similar processes. Listening effort was manipulated by parametrically varying spectral resolution (16- and 6-channel noise vocoding) and speech reception thresholds (SRT; 50% and 80%) while 19 young, normal-hearing adults performed a speech recognition task in noise. Results of off-line sentence scoring showed discrepancies between the target SRTs and the true performance obtained during the speech recognition task. For example, in the SRT80% condition, participants scored an average of 64.7%. Participants' true performance levels were therefore used for subsequent statistical modelling. Results showed that both measures appeared to be sensitive to changes in spectral resolution (channel vocoding), while pupil dilation only was also significantly related to their true performance levels (%) and task accuracy (i.e., whether the response was correctly or partially recalled). The two measures were not correlated, suggesting they each may reflect different cognitive processes involved in listening effort. This combination of findings contributes to a growing body of research aiming to develop an objective measure of listening effort.

Keywords: alpha power; listening effort; listening in noise; pupil dilation; spectral resolution; speech perception.

Figure 1.

Average pupil size over time for all trials and participants, for 16 - and 6-channel vocoding. The 0 s timepoint refers to the beginning of noise. All sentences finished at the 3.5 s timepoint. Shading represents ± 1 SE of the mean. The top panel represents the presentation protocol.

Figure 2.

Time–frequency representation of the EEG activity averaged across all participants, in the parietal region, for 16 - and 6-channel vocoding. The time–frequency representations are relative to the activity occurring during the 1 s of noise beginning at the 0 s timepoint. All sentences finished at the 3.5 s timepoint.

Figure 3.

Mean ± 1 SE of maximum pupil size and alpha power change relative to baseline, by SRT and channel vocoding.

Figure 4.

Pearson’s r correlation coefficients of pupil dilation and alpha power by participant and channel vocoding.