Audiovisual Perception of Sentence Stress in Cochlear Implant Recipients

Abstract

Background/Objectives: Sentence stress as part of linguistic prosody plays an important role for verbal communication. It emphasizes particularly important words in a phrase and is reflected by acoustic cues such as the voice fundamental frequency. However, visual cues, especially facial movements, are also important for sentence stress perception. Since cochlear implant (CI) recipients are limited in their use of acoustic prosody cues, the question arises as to what extent they are able to exploit visual features. Methods: Virtual characters were used to provide highly realistic but controllable stimuli for investigating sentence stress in groups of experienced CI recipients and typical-hearing (TH) peers. In addition to the proportion of correctly identified stressed words, task load was assessed via reaction times (RTs) and task-evoked pupil dilation (TEPD), and visual attention was estimated via eye tracking. Experiment 1 considered congruent combinations of auditory and visual cues, while Experiment 2 presented incongruent stimuli. Results: In Experiment 1, CI users and TH participants performed similarly in the congruent audiovisual condition, while the former were better at using visual cues. RTs were generally faster in the AV condition, whereas TEPD revealed a more detailed picture, with TH subjects showing greater pupil dilation in the visual condition. The incongruent stimuli in Experiment 2 showed that modality use varied individually among CI recipients, while TH participants relied primarily on auditory cues. Conclusions: Visual cues are generally useful for perceiving sentence stress. As a group, CI users are better at using facial cues than their TH peers. However, CI users show individual differences in the reliability of the various cues.

Keywords: cochlear implants; eye gaze; prosody; pupil dilation; reaction times; sentence stress; virtual reality.

Figure 1

Female and male virtual character used in the experiments. The green squares show the areas of interest defined for the left and right eye and the mouth as used for the eye-tracking analysis. Pink dots reveal an example of gaze fixation during a stimulus interval.

Figure 2

Proportion of correct stress identification for the different modification levels (1 = weak, 2 = intermediate, 3 = strong) and modalities (auditory-only, visual-only, audiovisual). (Left): CI recipients. (Right): TH participants. Modification levels in the A-only condition consider different changes in F0 for the CI recipients and the TH participants. The graph shows the mean and the standard error of the mean.

Figure 3

Audiovisual gain for the two study groups related to the modification level (1 = weak, 2 = intermediate, 3 = strong). The graph shows the mean and the standard error of the mean. CI = cochlear implant recipients, TH = typical-hearing participants.

Figure 4

Reaction times relative to the start of the sentence for the different modification levels (1 = weak, 2 = intermediate, 3 = strong) and modalities (auditory-only, visual-only, audiovisual). (Left): CI recipients. (Right): TH participants. The graphs show the mean and the standard error of the mean.

Figure 5

Change in pupil diameter relative to the baseline (interval 500 ms prior to sentence start) over the duration of the trial in the auditory-only, visual-only, and audiovisual modalities. The dashed vertical line indicates the sentence start. The horizontal bars indicate significant differences between the modalities, according to the BF10 factor. (Left): CI recipients, (Right): TH participants.

Figure 6

Dwell time related to the areas of interest (left and right eye, mouth) for the different modalities (A, V, AV) and study groups ((upper): CI recipients, (lower): TH subjects). The mean and the standard error of the mean are shown.

Figure 7

Modality dominance for the responses to the incongruent stimuli. The proportion of answers based on visual or auditory cues, respectively, is shown. Darker colored bars are given if the proportion of responses show a clear (binomial test, p ≤ 0.01) dominance of one of the modalities.

Figure 8

Change in pupil diameter relative to the baseline (interval 500 ms prior to sentence start) over the duration of the trial in the congruent and incongruent condition. The dashed vertical line indicates the sentence onset. The horizontal bars indicate significant differences between the modalities according to the BF10 factor. (Left): CI recipients, (Right): TH participants.

Figure 9

Change in pupil diameter relative to the baseline (interval 500 ms prior to sentence onset) over the duration of the trial in the auditory-only, visual-only, and audiovisual modalities. The dashed vertical line indicates the sentence onset. The horizontal bars indicate significant differences between the modalities according to the BF10 factor. (Left): CI recipients with visual-dominant behavior, (Right): CI recipients with auditory-dominant behavior.