Unity Assumption in Audiovisual Emotion Perception

Abstract

We experience various sensory stimuli every day. How does this integration occur? What are the inherent mechanisms in this integration? The "unity assumption" proposes a perceiver's belief of unity in individual unisensory information to modulate the degree of multisensory integration. However, this has yet to be verified or quantified in the context of semantic emotion integration. In the present study, we investigate the ability of subjects to judge the intensities and degrees of similarity in faces and voices of two emotions (angry and happy). We found more similar stimulus intensities to be associated with stronger likelihoods of the face and voice being integrated. More interestingly, multisensory integration in emotion perception was observed to follow a Gaussian distribution as a function of the emotion intensity difference between the face and voice-the optimal cut-off at about 2.50 points difference on a 7-point Likert scale. This provides a quantitative estimation of the multisensory integration function in audio-visual semantic emotion perception with regards to stimulus intensity. Moreover, to investigate the variation of multisensory integration across the population, we examined the effects of personality and autistic traits of participants. Here, we found no correlation of autistic traits with unisensory processing in a nonclinical population. Our findings shed light on the current understanding of multisensory integration mechanisms.

Keywords: Weak Central Coherence Theory; autistic traits; multisensory integration; semantic emotion perception; unity assumption.

FIGURE 1

Trial sequence in (A) the unisensory block; (B) the multisensory block. For demonstration purposes, a trial sequence with Strong Angry stimuli is illustrated. In the Weak condition, the duration of each frame was extended accordingly so the face sequence ended at 50% and stayed for 500 ms. In the Neutral condition, the presented face was always 0% Angry/Happy throughout the face sequence.

FIGURE 2

Summary of behavioral responses. (A) Emotion recognition accuracy for face (white bars) and voice (hatched bars); and (B) unisensory emotional intensity rating. *p < 0.05, **p < 0.01, ***p < 0.001. The error bars indicate standard errors of mean (SEM).

FIGURE 3

Audiovisual emotion intensity differences in multisensory integration. (A) Effect of audiovisual emotion intensity difference on judgment that audiovisual stimuli were similar in emotional intensity. Positive audiovisual emotional intensity differences indicate the emotional intensity of faces to be stronger than voice; negative differences indicate voices to be stronger than faces. The estimated parameters for the Gaussian distribution: a = 0.86, 95% CI = [0.73, 1.0]; b = 0.21, 95% CI = [0.1, 0.23]; c = 0.88, 95% CI = [0.72, 1.04]. (B) Face-voice emotional intensity absolute difference cut-off for differentiating Similar trials and Not Similar trials by emotion conditions. Optimal cut-offs were obtained from each participant using ROC analysis by maximizing the Youden index (J = Sensitivity + Specificity − 1). The error bars indicate standard errors of mean (SEM) in (A,B).

FIGURE 4

Sensory reliance in multisensory integration. (A) Sensory reliance score by emotion conditions. Positive values in sensory reliance score indicate a higher reliance on visual input, negative values indicating a higher reliance on auditory input during multisensory emotion perception. ^***p < 0.001. The error bars indicate standard errors of mean (SEM). (B) Relationship between AQ and sensory reliance score. Solid triangles and circles indicate the responses from the subjects with AQ ≥ 25.

FIGURE 5

Reliability-weighted multisensory integration model. Relationship between the difference in the audiovisual difference in emotion recognition accuracy and the sensory reliance scores in the Similar trials with (A) angry stimuli; and (B) happy stimuli. Positive values in sensory reliance score indicate a higher reliance on visual input, and negative values indicate a higher reliance on auditory input during multisensory emotion perception.