Emotion prediction as computation over a generative theory of mind

Abstract

From sparse descriptions of events, observers can make systematic and nuanced predictions of what emotions the people involved will experience. We propose a formal model of emotion prediction in the context of a public high-stakes social dilemma. This model uses inverse planning to infer a person's beliefs and preferences, including social preferences for equity and for maintaining a good reputation. The model then combines these inferred mental contents with the event to compute 'appraisals': whether the situation conformed to the expectations and fulfilled the preferences. We learn functions mapping computed appraisals to emotion labels, allowing the model to match human observers' quantitative predictions of 20 emotions, including joy, relief, guilt and envy. Model comparison indicates that inferred monetary preferences are not sufficient to explain observers' emotion predictions; inferred social preferences are factored into predictions for nearly every emotion. Human observers and the model both use minimal individualizing information to adjust predictions of how different people will respond to the same event. Thus, our framework integrates inverse planning, event appraisals and emotion concepts in a single computational model to reverse-engineer people's intuitive theory of emotions. This article is part of a discussion meeting issue 'Cognitive artificial intelligence'.

Keywords: affective computing; emotion; inverse planning; probabilistic generative model; social intelligence; theory of mind.

Figure 1.

Emotion prediction as inference over an intuitive Theory of Mind. Hypotheses about how human observers reason about others’ emotions can be formalized as probabilistic generative models. This reflects a hypothesis about observers’ intuitive theory of other people’s minds, not a scientific hypothesis about people’s actual emotions. (a) Implementation of the general hypothesis for the ‘Split or Steal’ game (a public one-shot Prisoner’s Dilemma). We treat observers’ emotion predictions as a function of their intuitive reasoning about how players will subjectively evaluate, or ‘appraise’, the game’s outcome. Observers predict a player’s emotions by inferring what preferences and beliefs motivated the player’s decision to Cooperate or Defect, and reason about how those preferences and beliefs would cause the player to emotionally react to the outcome of the game. The intuitive theories we test take the form of directed acyclic graphs, where arrows indicate the causal relationship between variables. Shaded nodes are observable variables and open nodes are latent variables. Round nodes are continuous variables, rectangular nodes are discrete variables. Nodes with a single border are random variables. The double border indicates that appraisals are calculated deterministically. Plans are shown with a partial border because they are not explicitly represented in this model. (b) Computational model of the intuitive theory. The model comprises three modules. Module (1) infers a joint distribution over preferences and beliefs given a player’s action via inverse planning. Module (2) computes appraisals based on how a player would evaluate the outcome of the game with respect to the inferred preferences and beliefs. Module (3) generates emotion predictions by transforming the computed appraisals. The probability density plots illustrate how observers’ prior belief about a player’s preference $\mathrm{P}(\omega )$ is updated based on the player’s action, and how the inferred preference $\mathrm{P}(\omega \mid a_1)$ is used to predict the player’s emotional reaction to the game’s outcome $\mathrm{P}(e\mid a_1,a_2)$.

Figure 2.

Inverse planning. Human observers were shown a player’s decision to cooperate (C) or defect (D) and judged the player’s likely preferences and belief. Model inversion yields a joint inference of the mental contents conditional on the players’ actions. Preference weights take continuous values between zero and one. Player 1’s belief about what player 2 will choose was rated on a 6-point confidence scale. Expectation shows the mean weight of each marginal distribution conditional on the player’s action ($a_1$).

Figure 3.

Generative structure. (a) Expectations and densities of the normalized computed appraisals. $\mathit{\Psi }$ is the matrix of the 19-dimensional appraisal vectors. The legend gives the relative payoffs to the players ($r_1,r_2$), e.g. when player 1 Cooperated and won nothing because player 2 Defected and won the whole pot, $a_1{a}_2\hspace{0.17em}(r_1,r_2)$: $\mathtt{\text{CD}}\hspace{0.17em}(0,1)$. Colour indicates the outcome of the games. (b) Example of a computed appraisal’s relationship to pot size. The $x$-axis shows the 24 pot sizes (non-parametric scaling), $y$-axis shows the loading on simulated players’ monetary utility prediction error, colour indicates density. (c) We learn a function ($\mathcal{g}$) that transforms computed appraisals ($\mathit{\Psi }$) into emotion predictions ($\mathcal{E}$) by scoring the empirical emotion vectors predicted for the GenericPlayers from the joint posterior over computed appraisals. To learn the transformation parameters, we leverage the expectations, as well as the hierarchical covariance structure of computed appraisals and of empirical emotion attributions. The result is a sparse weights matrix $\mathit{\beta }$. Expectations are shown in (a,d). Correlation matrices shown in (c) give the within-stimulus correlation (averaged within outcome) between computed appraisals (top) and between empirical emotion predictions (bottom). (d) Emotion predictions for the GenericPlayers. Circles show the expected intensity for each outcome, summing over pot sizes and the eight photos. Shading shows the density of judgements. $\mathcal{E}$ is the matrix of the 20-dimensional emotion prediction vectors. Each expectation (and associated distribution) reflects $n=1\hspace{0.17em}108$ judgements of $n=554$ observers.

Figure 4.

Appraisal structure of the intuitive theory of emotion. The $\beta$ weights of the transformation were learned based on the joint distribution of appraisals and the joint distribution of emotion predictions for the GenericPlayers. A Laplace prior over the $\mathit{\beta }$ weights induces a sparse solution. To learn the scale of the prior, we cross-validated on subsets of the SpecificPlayers and generated emotion predictions for the leftout players. Saturation indicates that the 99% CI does not overlap with zero. (a) Mean weights of the learned transformation. (b) These log-scale plots show the expectation, 95% and 99% CI of the weights learned for two example emotions, gratitude and embarrassment.

Figure 5.

Inferred emotion predictions. (a) Emotion predictions generated by the ComputedAppraisals model, averaged across players and pot sizes. (b) Example of how the model personalizes emotion predictions. Based on the GenericPlayers data, the model learned that envy is a function of appraisals derived from a player’s aversion to disadvantageous inequity ($\mathrm{base}\hspace{0.17em}DIA$). When a player appraises that he is in a more socially disadvantageous position than he expected to be (negative prediction error), the negative loading on $P{E}_{DIA}^{\mathrm{base}}$ translates to greater envy intensity. Similarly, when a player appraises that he would have been in a less disadvantageous position if his opponent had made the other choice (positive counterfactual), the positive loading on $CFa{2}_{DIA}^{\mathrm{base}}$ translates to greater envy intensity. Given that they chose to cooperate ($a_1=\mathtt{\text{C}}$), the model infers that the engineer cares more about not ending up in an inferior position (${\omega }_{\mathit{D}\mathit{I}\mathit{A}}^{\mathrm{base}}$) than the councilman. When the opposing player defects ($a_2=\mathtt{\text{D}}$), the model predicts greater intensity of envy for the engineer because he is inferred to have a stronger preference. Human observers similarly predicted that the engineer will experience more envy than the councilman in CD games. Photos of the players have been downsampled for the purpose of publication.

Figure 6.

Predicting specific player’s emotions. Human observers made preference and belief attributions to the 20 SpecificPlayers, based on a photo, brief description and decision, in the ‘Split or Steal’ game. (a) Based on what a SpecificPlayer was judged to care about and to expect, the models generated predictions of that player’s emotion reaction in 24 ‘Split or Steal’ games (four outcomes and eight pots). Bar colours in (b–e) correspond to the models in (a), and grey windows give the 95% bootstrap CI of the inter-rater reliability of the emotion predictions. (b) Concordance between predictions generated by the models and human observers for every emotion (collapsing across players, outcomes and pot sizes). (c) Overall fit the emotions observers predicted for the 20 SpecificPlayers. (d) The photos and descriptions of SpecificPlayers biased human observers’ judgements of the players’ motivations, expectations and emotional reactions. This plot shows how well the models were able to predict the bias in emotion predictions based on observers’ judgements of a player’s preferences and belief. Players are ordered based on how reliably observers’ emotion predictions differed from the emotions predicted for the GenericPlayers (grey windows). The model score gives the variance-scaled Pearson correlation. (e) Correlation between the relative difference predicted by the models and the relative difference in observers’ emotion predictions. (b,c,e) Each bar reflects a model’s performance based on $n=12\hspace{0.17em}096$ emotion predictions of $n=1\hspace{0.17em}512$ observers. ($d$) Each bar reflects a model’s performance based on a minimum of $n=579$ empirical predictions of all 20 emotions.