A dynamic, embodied paradigm to investigate the role of serotonin in decision-making

Abstract

Serotonin (5-HT) is a neuromodulator that has been attributed to cost assessment and harm aversion. In this review, we look at the role 5-HT plays in making decisions when subjects are faced with potential harmful or costly outcomes. We review approaches for examining the serotonergic system in decision-making. We introduce our group's paradigm used to investigate how 5-HT affects decision-making. In particular, our paradigm combines techniques from computational neuroscience, socioeconomic game theory, human-robot interaction, and Bayesian statistics. We will highlight key findings from our previous studies utilizing this paradigm, which helped expand our understanding of 5-HT's effect on decision-making in relation to cost assessment. Lastly, we propose a cyclic multidisciplinary approach that may aid in addressing the complexity of exploring 5-HT and decision-making by iteratively updating our assumptions and models of the serotonergic system through exhaustive experimentation.

Keywords: acute tryptophan depletion; adaptive agents; cognitive modeling; cost assessment; embodiment; game theory; human–robot interaction; serotonin.

FIGURE 1

Multidisciplinary paradigms. (A) Multidisciplinary paradigm for investigating the role of serotonin in decision-making and behavior. The model begins with base assumptions regarding neuromodulation, which are then used to develop an adaptive neural network model of cost and reward assessment. This network is embedded in an agent acting as a player in a game theoretic environment, alongside control conditions with set-strategy agents. These agents are both embodied in robotic players and simulated in computer-based games. The agents are used in both human and simulation experiments to assess the adaptive network’s ability to behave naturally, as well as the human subjects’ reactions to the adaptive agent compared to set-strategy agents. Human subject experiments under this paradigm can include acute tryptophan depletion (ATD) manipulations. The results from human and simulation experiments are then processed to determine the validity of any hypotheses developed at the outset, in addition to the appearance of interesting emergent behavior. (B) Cyclic, multidisciplinary paradigm. This model is a modified version of (A) with an added iterative component, as well as the inclusion of both fMRI (Tanaka et al., 2007; Yoshida et al., 2010; Crockett et al., 2012; Seymour et al., 2012) and genetics (Bevilacqua and Goldman, 2011; Hyde et al., 2011; Loth et al., 2011) components of human experimentation. The addition of an iterative component allows the results of previous studies conducted under the paradigm to be analyzed for possible areas of improvement in the model, which are then committed as alterations. The new node represents the following three modifications: (1) make new interpretations as to the role of serotonin in human subject behavior; (2) develop a new cognitive model based on human subject behavior; and (3) modify the adaptive neural network to create agents that reflect observed individual differences in human subject behavior. This paradigm allows for a constantly improving neural network model that is increasingly more able to fit the demands of studying decision-making and behavior.

FIGURE 2

Adaptive agent architectures. (A) General neural network architecture for Hawk-Dove and Chicken studies. The thick arrows represent all-to-all connections. The dotted arrows with the shaded oval represent modulatory plastic connections. Within the Action Neurons region, neurons with excitatory reciprocal connections are represented as arrow-ended lines, and neurons with reciprocal inhibitory connections are represented as dot-ended lines overlaid by a shaded oval, which denotes plasticity. (B) Actor-Critic schematic. The behavior of the adaptive agent used in the Stag Hunt experiment (Craig et al., 2013) was controlled by an Actor-Critic model. The model was comprised of three state tables – Actor, Cost Critic, and Reward Critic – which were updated with relevant information from the most recent turn in the current game and then used to determine whether the agent should hunt stag or hare on the next turn. The payoff information from the last turn was combined with the cost/reward value associated with the current state (determined by the locations of tokens on the board) from each respective critic using a delta-learning rule. These modified values were then used to update the corresponding state in the Actor table, which was used in a SoftMax function to generate probabilities for hunting stag and hare. Those probabilities were then used to determine the agent’s action on the upcoming turn.

FIGURE 3

Hawk-Dove game diagram. The game board included a 5 × 5 grid of squares, upon which a territory was marked and the human and neural agent players were placed. The color of the territory reflected the state of the players’ actions. In the Hawk-Dove, two players must compete for a territory, deciding either to be submissive (display) or aggressive (escalate), avoiding or risking injury in hopes of a larger payoff, respectively. © 2012 IEEE. Reprinted, with permission, from Asher et al. (2012a).

FIGURE 4

Chicken game diagram. Two toy cars, one driven by the human and one by the neural agent, were placed at opposite ends of a track. The cars started moving toward each other at the same speed and at the same time, at which point players must decide whether to conservatively swerve out of the way, but take a smaller payoff, or take the risk of a collision and continue straight ahead in hopes of a larger payoff. © 2012 IEEE. Reprinted, with permission, from Asher et al. (2012a).

FIGURE 5

Stag Hunt game environment. The game board included a 5 × 5 grid of spaces upon which the player (stick figure image), agent (robot image), stag (stag image), and hare (hare image) tokens resided. The screen included a button to start the experiment, the subject’s score for the round, the subject’s overall score for the experiment, the game number, a countdown to the start of the game, and a counter monitoring the game’s timeout. In the game of Stag Hunt, two players attempt to hunt a low-payoff hare alone, or attempt to cooperate with the other player to hunt a large payoff stag. © 2013 by Adaptive Behavior. Reprinted by Permission of SAGE from Craig et al. (2013).

FIGURE 6

Estimated group identities based on cognitive modeling results. Both plots show each subject’s likelihood to choose the aggressive action (escalate) for the two different conditions. Red and green dots correspond to subjects that showed a respective increased or decreased probability to escalate from their baselines. Error bars show the 95% Bayesian confidence interval of the posterior mean. The x-axes indicate subject numbers, which correspond to the same subjects in the two plots. The y-axes show the Bayesian model’s mean output indicating group affiliation with respect to the subject’s likelihood to escalate, relative to their independently determined baselines. The y-axis value of 1 indicates a strong likelihood of decreasing choices to escalate relative to their baseline level for the conditions, whereas the value of 2 indicates a strong likelihood of increasing choices to escalate relative to their baseline level for the conditions. The group identities were estimated based on: (A) the influence tryptophan depletion had on subjects’ choices for aggressive actions (Escalation, Tryptophan), and (B) the influence an embodied agent had on subjects’ choices for aggressive actions (Escalation, Robot). Cognitive Science Conference and published in the Proceedings (COGSCI 2012, Sapporo, JP) from Asher et al. (2012b).