Voice emotion perception and production in cochlear implant users

Abstract

Voice emotion is a fundamental component of human social interaction and social development. Unfortunately, cochlear implant users are often forced to interface with highly degraded prosodic cues as a result of device constraints in extraction, processing, and transmission. As such, individuals with cochlear implants frequently demonstrate significant difficulty in recognizing voice emotions in comparison to their normal hearing counterparts. Cochlear implant-mediated perception and production of voice emotion is an important but relatively understudied area of research. However, a rich understanding of the voice emotion auditory processing offers opportunities to improve upon CI biomedical design and to develop training programs benefiting CI performance. In this review, we will address the issues, current literature, and future directions for improved voice emotion processing in cochlear implant users.

Keywords: Cochlear implant; Speech prosody; Voice emotion; Voice emotion perception; Voice emotion production.

Figure 1

17 school-aged children were asked to discriminate sinusoidal amplitude modulation rate (AMR) of broadband noise and to discriminate the fundamental frequency (F0) of broadband sine-phase harmonic complexes (Deroche, et al., 2012). Children who had adult-like sensitivity were not necessarily the oldest listeners. For example, the youngest research subject was 6.5 years old and demonstrated a standardized threshold of only 1.08 semitones for AMR discrimination and 12 cents for F0 discrimination, suggesting that children’s sensitivity to pitch (regardless of the underlying cue) does not systematically improve beyond 6 years of age.

Figure 2

Percent correct scores for CI users (left) and NH listeners (right) on four subtests of PEPS-C, including turn-end, affect, chunking, and contrastive stress (Kalathottukaren, et al., 2015). As indicated by the asterisk, chunking was the only task for which CI users did not perform significantly worse than NH listeners.

Figure 3

Mean emotion recognition scores for adult NH, adult CI, child NH, and child CI study groups across full spectrum speech and speech presented with three levels of spectral degradation (Chatterjee, et al., 2015). Child and adult CI users’ performance were similar to one another’s and comparable to adult NH’s performance in 8-channel degraded speech.

Figure 4

Cochlear implant (CI) versus normal hearing (NH) group mean proportions of question judgements as a function of an acoustic dimension (F₀ height, F₀ contour, peak intensity ratio, and duration ratio) (Peng, et al., 2012). The NH group was also subject to different listening conditions (full-spectrum, 8-, and 4-channel conditions) as depicted by different symbols.

Figure 5

Accuracy rate in identification of neutral, angry, and happy prosody in NH listeners and CI users using ACE and MP3000 strategies (Agrawal, et al., 2013).

Figure 6

Graphical representation of improvements in question-statement discrimination (as measured by the PEPS-C test) in CI users after participation in melodic contour training (Yhun Lo, et al., 2015). Two programs of melodic contour training were administered. The “Interval” program incorporated manipulations of note intervals, whereas the “Duration” program changed the note durations, in order to make the tests more challenging. The dashed line represents chance performance.