Cortical systems for the recognition of emotion in facial expressions

Abstract

This study is part of an effort to map neural systems involved in the processing of emotion, and it focuses on the possible cortical components of the process of recognizing facial expressions. We hypothesized that the cortical systems most responsible for the recognition of emotional facial expressions would draw on discrete regions of right higher-order sensory cortices and that the recognition of specific emotions would depend on partially distinct system subsets of such cortical regions. We tested these hypotheses using lesion analysis in 37 subjects with focal brain damage. Subjects were asked to recognize facial expressions of six basic emotions: happiness, surprise, fear, anger, disgust, and sadness. Data were analyzed with a novel technique, based on three-dimensional reconstruction of brain images, in which anatomical description of surface lesions and task performance scores were jointly mapped onto a standard brain-space. We found that all subjects recognized happy expressions normally but that some subjects were impaired in recognizing negative emotions, especially fear and sadness. The cortical surface regions that best correlated with impaired recognition of emotion were in the right inferior parietal cortex and in the right mesial anterior infracalcarine cortex. We did not find impairments in recognizing any emotion in subjects with lesions restricted to the left hemisphere. These data provide evidence for a neural system important to processing facial expressions of some emotions, involving discrete visual and somatosensory cortical sectors in right hemisphere.

Fig. 1.

Performance scores on recognition of facial expression for all subjects. Pearson correlations between a brain-damaged subject’s ratings and normal ratings are shown for each subject and for each emotion category used in the task. The recognition of fearful faces is impaired in the largest number of subjects, and the recognition of happy faces is never impaired.

Fig. 2.

Mean performance scores on recognition of facial expressions for subjects with left (a) and right (b) hemisphere lesions. Performance scores are correlations of a subject’s rating of a facial expression with the mean ratings given by normal controls. The unweighted mean scores were calculated for that emotion on which each subject performed the worst, so as to give a lower limit to the ability to process emotions in general. Thus, if a subject was impaired in recognizing any of the six emotions, he would be impaired on this measure. Composite extents of lesions for all subjects, together with their mean scores on their worst individual emotion performances, are shown on the lateral and mesial surfaces of the hemispheres. The number of subjects sharing a lesion locus is encoded by the saturation (fainter colors correspond to small numbers of subjects, andstronger colors correspond to larger numbers of subjects), and the mean score is encoded by the color of each pixel (yellow and red hues correspond to more impaired performances, and blue and green hues correspond to more normal performances), as indicated in the scale. The figure shows that there were no impairments in recognizing any facial expressions among subjects with left hemisphere lesions, but that some subjects with lesions in regions of the right hemisphere were impaired.

Fig. 3.

Unweighted mean performance scores on recognition of specific facial expressions for subjects with right hemisphere lesions. Unweighted mean correlation scores are shown for each emotion for all subjects with lesions in the lateral (left) or mesial (right) aspects of the right hemisphere. Pixel attributes are as in Figure 2; hue corresponds to the mean score of all the subjects who had a lesion that included a given pixel location. The recognition of fear was most impaired in subjects with lesions in right hemisphere. However, there are also more subtle differences among the other emotions. Happiness was recognized entirely normally (green) with respect to lesions at any location, whereas lesions that included a region within the supramarginal gyrus resulted in a somewhat impaired (purple) recognition of sad faces. Lesions restricted to the anterior and inferior temporal cortex did not result in impairments in recognizing any emotion (green–blue of this region in all images).

Fig. 4.

Anatomical regions involved in the recognition of fear. a, Anatomical overlap of lesions of subject groups. We calculated overlaps only for the lateral (left) and mesial (right) aspects of the right hemisphere, because all subjects with left hemisphere lesions were normal on our task. The top panel shows the overlap of the lesions of all subjects. Bottom panels show the overlap of the lesions of all subjects whose score on recognition of fear (equivalent to their lowest score on any emotion) was less than a given cut-off value, indicated on the figure. The maximal overlap of the lesions of those subjects with the most impaired scores was in the right inferior parietal cortex and in the right infracalcarine cortex. It should be noted that a single impaired subject whose lesion encompassed both mesial and lateral right occipital cortex is visible in all panels. Although this subject appears on the lateral views, we think it likely that his impaired performance in fact results from the inclusion of right mesial occipital sectors, which he shares in common with other impaired subjects. b, Weighted mean performance scores on recognition of facial expressions of fear for subjects with lesions of the right hemisphere. In this figure, pixel hue corresponds to the mean of subjects’ weighted scores, such that more normal scores contribute more to the mean than do more impaired scores; see Materials and Methods for details. In the lateral aspect of the right hemisphere (left), there is a hot-spot in the right supramarginal and right posterior superior temporal gyri. In the mesial aspect of the right hemisphere (right) there is a hot-spot in the posterior sector of the right anterior infracalcarine cortex. Subjects whose lesions included the hot-spot region were the most impaired in their recognition of fear. Pixel hue and saturation are encoded as in the scale to Figure 2. Convergent results were obtained in a and b.

Fig. 5.

Performance scores of the recognition of fear are plotted against four independent variables: performance IQ, age, performance on the Benton facial discrimination test (raw score), and score on the Beck Depression Inventory. Scores on the recognition of fear correlated only with performance IQ and age; no other neuropsychological variable covaried significantly.

Fig. 6.

Summary of findings. Recognition of facial expressions of emotion is most impaired by lesions in specific right hemisphere locations. Our data point to two loci (black circles), in right inferior parietal cortex on the lateral surface (supramarginal gyrus, hatched region) and in right infracalcarine cortex on the mesial surface.