Emotional speech processing at the intersection of prosody and semantics

Abstract

The ability to accurately perceive emotions is crucial for effective social interaction. Many questions remain regarding how different sources of emotional cues in speech (e.g., prosody, semantic information) are processed during emotional communication. Using a cross-modal emotional priming paradigm (Facial affect decision task), we compared the relative contributions of processing utterances with single-channel (prosody-only) versus multi-channel (prosody and semantic) cues on the perception of happy, sad, and angry emotional expressions. Our data show that emotional speech cues produce robust congruency effects on decisions about an emotionally related face target, although no processing advantage occurred when prime stimuli contained multi-channel as opposed to single-channel speech cues. Our data suggest that utterances with prosodic cues alone and utterances with combined prosody and semantic cues both activate knowledge that leads to emotional congruency (priming) effects, but that the convergence of these two information sources does not always heighten access to this knowledge during emotional speech processing.

Figure 1. Examples of prime and target stimuli presented in the pre-semantic and post-semantic conditions.

For expository purposes, the pre-semantic condition shows a congruent “YES” trial and corresponding “NO” trial, whereas the post-semantic condition shows an incongruent “YES” trial and “NO” trial.

Figure 2. Effects of prime type on response latencies.

Effects of congruent, incongruent, and neutral voice primes on facial affect decision latencies (in milliseconds) made immediately prior to the emotional semantic word onset (pre-semantic condition) and immediately following the emotional semantic word (post-semantic condition). Error bars display the standard error of the means.