Neural Representations of Emotions in Visual, Auditory, and Modality-Independent Regions Reflect Idiosyncratic Conceptual Knowledge

Abstract

Growing evidence suggests that conceptual knowledge influences emotion perception, yet the neural mechanisms underlying this effect are not fully understood. Recent studies have shown that brain representations of facial emotion categories in visual-perceptual areas are predicted by conceptual knowledge, but it remains to be seen if auditory regions are similarly affected. Moreover, it is not fully clear whether these conceptual influences operate at a modality-independent level. To address these questions, we conducted a functional magnetic resonance imaging study presenting participants with both facial and vocal emotional stimuli. This dual-modality approach allowed us to investigate effects on both modality-specific and modality-independent brain regions. Using univariate and representational similarity analyses, we found that brain representations in both visual (middle and lateral occipital cortices) and auditory (superior temporal gyrus) regions were predicted by conceptual understanding of emotions for faces and voices, respectively. Additionally, we discovered that conceptual knowledge also influenced supra-modal representations in the superior temporal sulcus. Dynamic causal modeling revealed a brain network showing both bottom-up and top-down flows, suggesting a complex interplay of modality-specific and modality-independent regions in emotional processing. These findings collectively indicate that the neural representations of emotions in both sensory-perceptual and modality-independent regions are likely shaped by each individual's conceptual knowledge.

Keywords: conceptual knowledge; emotion; facial expressions; modality‐independent; vocal expressions.

FIGURE 1

Overview of Experiment 2. Participants performed a pre‐scan conceptual similarity rating behavioral task, completed a 1‐h fMRI scan, and concluded with a post‐scan emotion categorization and dimensional rating behavioral tasks.

FIGURE 2

Experiment 1 results. (a) Average conceptual dissimilarity matrix. (b) Distributions of conceptual dissimilarity. (c) Multidimensional scaling results. (d) Hierarchical clustering results.

FIGURE 3

Conceptual knowledge was related to emotion categorization. (a) Average emotion categorization accuracies with individual participant data overlaid. (b) Overall emotion categorization confusion matrix. (c) Average pre‐scan conceptual dissimilarity matrix. (d) Significant correlation between the average conceptual dissimilarity and the overall emotion categorization performances.

FIGURE 4

Lower‐dimensional representation of conceptual knowledge was comparable to explicit valence and arousal ratings. (a) Average valence ratings with overlaid individual participant data. (b) Average arousal ratings with individual participant data overlaid. (c) Post‐scan explicit valence and arousal ratings of facial and vocal expressions. (d) Lower‐dimensional representation of pre‐scan conceptual knowledge ratings revealed by multidimensional scaling.

FIGURE 5

Univariate activation results. (a) Brain activations associated with facial emotion processing. (b) Brain activations associated with vocal emotion processing. (c) Brain activations associated with emotion processing independent of face and voice modality.

FIGURE 6

RSA results. (a) Brain representations associated with facial emotion processing reflected individual variability in emotion‐concept knowledge. (b) Brain representations associated with vocal emotion processing reflected individual variability in emotion‐concept knowledge. Note: The figure shows threshold‐free cluster‐enhanced z‐maps thresholded at a z‐score of 1.65, corresponding to p < 0.05, one‐tailed (corrected for multiple comparisons).

FIGURE 7

DCM results with values indicating connectivity strength (in Hz). (a) An illustration of three regions and emotion‐independent connectivity between regions for facial emotion processing, thresholded at a posterior probability (Pp) of > 0.95. mOcc_l = left middle occipital cortex; InfOcc_r = right inferior occipital cortex; STS_r = right STS. (b) An illustration of three regions and emotion‐independent connectivity between regions for vocal emotion processing, thresholded at Pp > 0.95. STG_l = left STG; STG_r = right STG; STS_r = right STS. (c) Emotion‐specific modulatory connectivity between three regions for facial emotion processing. Gray color represents connections with Pp < 0.95. (d) Emotion‐specific modulatory connectivity between three regions for vocal emotion processing. Gray color represents connections with Pp < 0.95. Green and purple arrows indicate positive and negative effective connectivity, respectively.