Supramodal representations of perceived emotions in the human brain

Abstract

Basic emotional states (such as anger, fear, and joy) can be similarly conveyed by the face, the body, and the voice. Are there human brain regions that represent these emotional mental states regardless of the sensory cues from which they are perceived? To address this question, in the present study participants evaluated the intensity of emotions perceived from face movements, body movements, or vocal intonations, while their brain activity was measured with functional magnetic resonance imaging (fMRI). Using multivoxel pattern analysis, we compared the similarity of response patterns across modalities to test for brain regions in which emotion-specific patterns in one modality (e.g., faces) could predict emotion-specific patterns in another modality (e.g., bodies). A whole-brain searchlight analysis revealed modality-independent but emotion category-specific activity patterns in medial prefrontal cortex (MPFC) and left superior temporal sulcus (STS). Multivoxel patterns in these regions contained information about the category of the perceived emotions (anger, disgust, fear, happiness, sadness) across all modality comparisons (face-body, face-voice, body-voice), and independently of the perceived intensity of the emotions. No systematic emotion-related differences were observed in the overall amplitude of activation in MPFC or STS. These results reveal supramodal representations of emotions in high-level brain areas previously implicated in affective processing, mental state attribution, and theory-of-mind. We suggest that MPFC and STS represent perceived emotions at an abstract, modality-independent level, and thus play a key role in the understanding and categorization of others' emotional mental states.

Figure 1.

Schematic overview of the design. Each run consisted of 3 blocks of 12 trials each. Blocks differed in the type of stimuli presented (bodies, faces, or voices). On each trial, one of five different emotions could be expressed in a given modality. These trials were presented in random order within each block, whereas the order of blocks was counterbalanced across runs. Each trial consisted of a 2 s fixation cross, followed by a 3 s stimulus (movie or sound clip), a 1 s blank screen, and then a 2.5 s response window. After the presentation of each stimulus, participants were asked to rate the intensity of the perceived emotion on a 3-point scale. Not drawn to scale.

Figure 2.

Average intensity ratings for each experimental condition. ang, Anger; dis, disgust; fea, fear; hap, happiness; sad, sadness.

Figure 3.

Results of a whole-brain searchlight analysis showing clusters with significant emotion-specific activity patterns across modality. Similarity of activity patterns was expressed as a correlation value, with higher correlations indicating higher similarity. Patterns were generally more similar within emotion categories (green bars) than between emotion categories (blue bars). Top row: Cluster in MPFC; peak (x, y, z): 11, 48, 17; t₁₇ = 6.0, p = 0.00001, and average Fisher-transformed correlation values in MPFC cluster for the three cross-modality comparisons. Correlations were higher for within- than between-emotion comparisons for all three modality combinations (p < 0.05, for all tests). Bottom row: Cluster in left STS; peak (x, y, z): −47, −62, 8, t₁₇ = 6.4, p < 0.00001, and average Fisher-transformed correlation values in STS cluster for the three cross-modality comparisons. Correlations were higher for within- than between-emotion comparisons for all three modality combinations (p < 0.05, for all tests). ang, Anger; dis, disgust; fea, fear; hap, happiness; sad, sadness; all, average across the five emotions. Error bars indicate within-subject SEM.

Figure 4.

Mean activation (parameter estimates) in MPFC (left) and STS (right) searchlight clusters for each experimental condition. Abbreviations are as in Figure 2.