Face-selective regions differ in their ability to classify facial expressions

Abstract

Recognition of facial expressions is crucial for effective social interactions. Yet, the extent to which the various face-selective regions in the human brain classify different facial expressions remains unclear. We used functional magnetic resonance imaging (fMRI) and support vector machine pattern classification analysis to determine how well face-selective brain regions are able to decode different categories of facial expression. Subjects participated in a slow event-related fMRI experiment in which they were shown 32 face pictures, portraying four different expressions: neutral, fearful, angry, and happy and belonging to eight different identities. Our results showed that only the amygdala and the posterior superior temporal sulcus (STS) were able to accurately discriminate between these expressions, albeit in different ways: the amygdala discriminated fearful faces from non-fearful faces, whereas STS discriminated neutral from emotional (fearful, angry and happy) faces. In contrast to these findings on the classification of emotional expression, only the fusiform face area (FFA) and anterior inferior temporal cortex (aIT) could discriminate among the various facial identities. Further, the amygdala and STS were better than FFA and aIT at classifying expression, while FFA and aIT were better than the amygdala and STS at classifying identity. Taken together, our findings indicate that the decoding of facial emotion and facial identity occurs in different neural substrates: the amygdala and STS for the former and FFA and aIT for the latter.

Trial registration: ClinicalTrials.gov NCT01087281.

Keywords: Amygdala; Emotional faces; STS; SVM; fMRI.

Figure 1

Schematic showing the main task (A) and the stimulus set used for the main task. A) The trial began with the presentation of a face image for 300 ms with a colored fixation cross centered on the image, followed by a white fixation cross centered on the screen for the rest of the trial (7700 ms). The colored fixation-cross appeared when a face image appeared and changed to a white fixation cross when the face image disappeared. Each trial lasted a fixed time duration of 8 seconds. There were 32 face trials in one run and each run lasted 4 min 32 seconds. In the MRI scanner, subjects viewed the visual stimuli projected onto a screen and pressed the left button when the fixation cross changed from white to red, and the right button when the fixation cross changed from white to green. B) The face stimulus set contained 32 faces, which belonged to four categories of expression: neutral, fearful, angry, and happy, each consisting of eight different facial identities. Half of the stimuli were female and half were male. Each face image was presented once in each run. The order of the face images was randomized across runs.

Figure 2

Example of localization of face-selective regions by contrasting the fMRI response evoked by face stimuli compared to the response evoked by object stimuli (face > object) for a single subject. This contrast identified five face-selective regions in the right hemisphere: the dorsal-lateral part of the amygdala; anterior inferior temporal cortex - aIT; fusiform face area -FFA; posterior superior temporal sulcus - posterior STS and occipital face area – OFA. Primary visual cortex - V1 was selected as a control region of interest.

Figure 3

The decoding performance (average z-normalized accuracy rate across subjects) of each face-selective region of interest (ROI; panels A - E) as well as V1 (panel F) for classification of facial expression. For each ROI, the decoding was performed by using one- versus-three SVM to classify each category of facial expression (neutral, fearful, angry and happy) relative to all other facial expression categories.

Figure 4

Hierarchical classification analysis of facial expression in the amygdala and posterior STS. The hierarchical structure was calculated by applying one-versus-all SVM classification at each level of the binary tree architecture. In the amygdala (panels A–C), the classification of fearful and non-fearful faces was at the top of the hierarchy and showed significant accuracy of discrimination between the two (A); the classification of neutral and emotional faces (angry and happy) was at the second level and showed significant accuracy of discrimination (B); the classification of happy and angry faces was at the bottom of the hierarchy but did not show significant accuracy of discrimination (C). In contrast, in STS (panels D–F), the classification of neutral and emotional faces (fearful, angry and happy) was at the top of the hierarchy and showed significant accuracy of discrimination (D); the classification of positive (happy) and negative (fearful and angry) expressions was at the second level and showed significant accuracy of discrimination (E); the classification of angry and fearful faces was at the bottom of the hierarchy and did not show significant accuracy of discrimination (F).

Figure 5

Percent signal change of the fMRI response amplitude for each category of facial expression (neutral, fearful, angry and happy) in face-selective ROIs (panels A–E) and V1 (panel F) calculated using the conventional univariate analysis approach.

Figure 6

The decoding performance (average of z-normalized accuracy rate across subjects) of each face-selective ROIs (panels A–E) as well as V1 (panel F) for facial identity classification. For each ROI, the decoding was performed by using one-versus-three SVM to classify each individual face relative to the other three individuals of the same gender. Dark gray bars indicate male faces; light gray bars indicate female faces.

Figure 7

Average percent signal change of the fMRI response amplitude for each face identity in face- selective ROIs (panels A–E) and V1 (panel F) calculated using the conventional univariate analysis approach.

Figure 8

Summary of the hierarchical classification structure in amygdala and posterior STS. Solid black lines outline the facial expressions that can successfully be classified; dashed black lines outline the facial expressions that cannot be classified.