Conceptual knowledge predicts the representational structure of facial emotion perception

Abstract

Recent theoretical accounts argue that conceptual knowledge dynamically interacts with processing of facial cues, fundamentally influencing visual perception of social and emotion categories. Evidence is accumulating for the idea that a perceiver's conceptual knowledge about emotion is involved in emotion perception, even when stereotypic facial expressions are presented in isolation^1-4. However, existing methods have not allowed a comprehensive assessment of the relationship between conceptual knowledge and emotion perception across individuals and emotion categories. Here we use a representational similarity analysis approach to show that conceptual knowledge predicts the representational structure of facial emotion perception. We conducted three studies using computer mouse-tracking⁵ and reverse-correlation⁶ paradigms. Overall, we found that when individuals believed two emotions to be conceptually more similar, faces from those categories were perceived with a corresponding similarity, even when controlling for any physical similarity in the stimuli themselves. When emotions were rated conceptually more similar, computer-mouse trajectories during emotion perception exhibited a greater simultaneous attraction to both category responses (despite only one emotion being depicted; studies 1 and 2), and reverse-correlated face prototypes exhibited a greater visual resemblance (study 3). Together, our findings suggest that differences in conceptual knowledge are reflected in the perceptual processing of facial emotion.

Fig. 1 |. Mouse-tracking effects.

On each trial of the mouse-tracking tasks, subjects were presented with a face stimulus and categorized it as one of two emotion categories, one of which corresponded to the posed emotion in the stimulus. MD of subjects’ hand trajectories towards an unselected response is an index of how much that category was co-activated during perception. Complete mouse-tracking results for studies 1 and 2 are presented in Figs. 2 and 3 and Supplementary Fig. 3. To provide a visual example of the pattern of results for one category pair (Sad–Surprised), mean mouse trajectories during Sad versus Surprised trials in study 2 are depicted, separately for subjects with low and high conceptual similarity between Sad and Surprised categories (using median split). Trajectories are averaged across both Sad and Surprised responses and only arbitrarily depicted here as selecting the Surprised response. Subjects with high conceptual similarity exhibited a greater simultaneous attraction to select both emotion categories (even though only one was depicted on a given trial), manifesting in the mean trajectories and MD. Note that median split is only used here for visualization purposes; all analyses were conducted treating conceptual overlap as a continuous variable.

Fig. 2 |. Average conceptual and perceptual DMs for studies 1–3 and schematic of the analytic approach.

In all studies, conceptual similarity and perceptual similarity were assessed for all pairwise emotion combinations (for example, Anger–Disgust). Each subject’s 15 unique emotion category pairs (unique values under the diagonal) for both conceptual structure and perceptual structure were vectorized and submitted to multilevel regression analyses, predicting perceptual similarity values from conceptual similarity values. Average conceptual and perceptual similarity structure (dissimilarity matrices (DMs)) are shown for all studies (study 1, N = 100; study 2, N = 91; study 3, N = 368). Average DMs for the two additional measures of perceptual similarity collected in study 3 are presented in Supplementary Fig. 1. Note that the average DMs for study 3 are presented for illustrative purposes only, as each cell under the diagonal included a different set of subjects. Due to the length of the task in study 3, each participant was randomly assigned to a different condition, where each condition was a given emotion category pair (for example, Anger–Disgust). The overall hypothesis was that a greater conceptual similarity between any two emotion categories (for example, Anger–Disgust) would correspond to a greater bias to perceive those emotions more similarly, measured by a simultaneous attraction to select both emotions during face perception with mouse tracking (studies 1 and 2) and a greater resemblance in estimated visual prototypes for the two emotions using reverse correlation (study 3).

Fig. 3 |. Multilevel regression results for studies 1–3.

For illustrative purposes only, each subject’s relationship between conceptual and perceptual similarity values are plotted as linear slopes (using ordinary least squares). Actual analyses were conducted using GEE multilevel regression. Individual subject slopes are not depicted for study 3 (instead, all data points are depicted) due to study 3’s design in which no individual subject completed all experimental conditions. A positive relationship between conceptual and perceptual similarity was observed across studies. In studies 1 (N = 100) and 2 (N = 91), perceptual similarity was measured as average MD towards the unselected category response on mouse-tracking trials with the two categories in question as response options (for example, Angry–Disgusted). In study 3 (N = 368), perceptual similarity was measured through independent ratings of reverse-correlated prototype faces from each category. Mean slopes are shown in blue.

Fig. 4 |. Reverse-correlation effects.

Reverse-correlation allows an estimation of each subject’s visual prototype for a given emotion category. Complete reverse-correlation results for study 3 are presented in Fig. 3. To provide a visual example of the pattern of results for one category pair (Angry–Sad), reverse-correlated classification images are depicted, separately for subjects in tertiles of high, average and low conceptual similarity between the Angry and Sad categories. Subjects with higher conceptual similarity between two categories exhibited a greater resemblance in the appearance of their classification images for those two categories, as assessed by independent ratings of emotion category, independent ratings of subjective perceptual similarity and an objective measure of pixel-based similarity of the classification images themselves.