Musical Emotion Categorization with Vocoders of Varying Temporal and Spectral Content

Abstract

While previous research investigating music emotion perception of cochlear implant (CI) users observed that temporal cues informing tempo largely convey emotional arousal (relaxing/stimulating), it remains unclear how other properties of the temporal content may contribute to the transmission of arousal features. Moreover, while detailed spectral information related to pitch and harmony in music - often not well perceived by CI users- reportedly conveys emotional valence (positive, negative), it remains unclear how the quality of spectral content contributes to valence perception. Therefore, the current study used vocoders to vary temporal and spectral content of music and tested music emotion categorization (joy, fear, serenity, sadness) in 23 normal-hearing participants. Vocoders were varied with two carriers (sinewave or noise; primarily modulating temporal information), and two filter orders (low or high; primarily modulating spectral information). Results indicated that emotion categorization was above-chance in vocoded excerpts but poorer than in a non-vocoded control condition. Among vocoded conditions, better temporal content (sinewave carriers) improved emotion categorization with a large effect while better spectral content (high filter order) improved it with a small effect. Arousal features were comparably transmitted in non-vocoded and vocoded conditions, indicating that lower temporal content successfully conveyed emotional arousal. Valence feature transmission steeply declined in vocoded conditions, revealing that valence perception was difficult for both lower and higher spectral content. The reliance on arousal information for emotion categorization of vocoded music suggests that efforts to refine temporal cues in the CI user signal may immediately benefit their music emotion perception.

Keywords: arousal; cochlear implants; music emotion perception; valence; vocoders.

Figure 1.

Example stimuli spectrograms shown in non-vocoded and vocoded conditions. Fear, a high-arousal emotion, shows more energy at frequencies across the spectrum in all conditions. Sadness, in comparison, is lower arousal and accordingly displays lower spectral energy across frequencies, especially compared to higher frequencies in the fear condition. Compared to non-vocoded spectrograms, vocoders with sinewave carriers contain energy more localized to channel frequencies whereas in the noise carrier, energy is smeared across multiple frequencies.

Figure 2.

Participants were presented with the scenario that they were an astronaut on an alien planet who was helping aliens to understand earth music. (Illustration by Kristin Hrycyk, image published under the CC BY NC 4.0 license, https://creativecommons.org/licenses/by-nc/4.0/).

Figure 3.

Sensitivity in perception of presented emotion categories (d’) as a function of vocoder manipulations (x-axis), and shown for different emotions (color and symbol). The horizontal line in the boxplot shows the median sensitivity across participants. The box extends from the 25^th to the 75^th percentile, and the whiskers extend to the value most remote from the median within 1.5 times the interquartile range. Individual data points are overlaid on top of the boxplots. d' = 0 denotes the chance level.

Figure 4.

Confidence ratings per vocoder. In all except the non-vocoded condition, confidence was reported higher as sensitivity increased.

Figure 5.

Confusion matrices for emotion categorization shown for each vocoder condition. The size and color of each dot is proportional to the number of relevant responses. The presented emotion categories are listed on the x-axis, while the responded emotion categories are listed on the y-axis.

Figure 6.

Relative transmission in percent of the total available information, shown as a function of feature (hatched vs. solid) and vocoder (x-axis). The sinewave and noise carriers are further marked by color (sinewave: blue; noise: red). The signification of the boxplot details is identical to that of Figure 3.