The nucleus accumbens as a nexus between values and goals in goal-directed behavior: a review and a new hypothesis

Abstract

Goal-directed behavior is a fundamental means by which animals can flexibly solve the challenges posed by variable external and internal conditions. Recently, the processes and brain mechanisms underlying such behavior have been extensively studied from behavioral, neuroscientific and computational perspectives. This research has highlighted the processes underlying goal-directed behavior and associated brain systems including prefrontal cortex, basal ganglia and, in particular therein, the nucleus accumbens (NAcc). This paper focusses on one particular process at the core of goal-directed behavior: how motivational value is assigned to goals on the basis of internal states and environmental stimuli, and how this supports goal selection processes. Various biological and computational accounts have been given of this problem and of related multiple neural and behavior phenomena, but we still lack an integrated hypothesis on the generation and use of value for goal selection. This paper proposes an hypothesis that aims to solve this problem and is based on this key elements: (a) amygdala and hippocampus establish the motivational value of stimuli and goals; (b) prefrontal cortex encodes various types of action outcomes; (c) NAcc integrates different sources of value, representing them in terms of a common currency with the aid of dopamine, and thereby plays a major role in selecting action outcomes within prefrontal cortex. The "goals" pursued by the organism are the outcomes selected by these processes. The hypothesis is developed in the context of a critical review of relevant biological and computational literature which offer it support. The paper shows how the hypothesis has the potential to integrate existing interpretations of motivational value and goal selection.

Keywords: amygdala; goal selection; goal-directed Behavior; hippocampus; novelty; nucleus accumbens; prefrontal cortex; value.

Figure 1

The major associations and processes behind instrumental habitual behavior, goal-directed behavior, Pavlovian processes, and their relations (here we describe only those relevant for this work). At the top, the diagram shows the systems processing the hedonic impact and the incentive value of stimuli, the latter important for the assignment of value to appetitive/aversive stimuli involved in goal-directed processes. The middle of the diagram shows the loop of processes involving goal-directed behavior; here the action representations are associated with outcome representations (instrumental contingency) and then these outcomes are attributed incentive value. In this way, outcomes can trigger the execution of motor responses that lead to them. The bottom of the diagram refers to Pavlovian processes, with the core association between conditioned stimuli (CS) to unconditioned stimuli (US). These have a certain value depending on the animal's internal states. Pavlovian processes can directly trigger unlearned behaviors (e.g., as in conditioned approach experiments) or influence the performance of instrumental behaviors (Pavlovian-Instrumental Transfer—PIT). The diagram also represents the formation of habits (S-R behaviors) as direct associations between stimuli (CS) and motor responses. Reprinted from Cardinal et al. (2002a), Copyright 2002, with permission from Elsevier.

Figure 2

Diagram of the rat brain illustrating a proposal on the role of DLS to select habits, of DMS to drive goal-directed behavior, and of NAcc to assign vigor to the performance of selected behaviors and to trigger ancillary behaviors such as orienting and approaching. Reprinted with permission (Gruber and McDonald, 2012).

Figure 3

Internal anatomy of a circuit loop through basal ganglia and cortex. GPe: globus-pallidus, external compartment; GPi: globus-pallidus, internal compartment; SNpr: substantia nigra pars reticulata; STN: sub-thalamic nucleus.

Figure 4

The three main striato-cortical loops involving different territories of BG and cortical areas. Reprinted by permission from Macmillan Publishers Ltd: Nature Reviews Neuroscience (Yin and Knowlton, 2006), copyright 2006.

Figure 5

A typical reinforcement learning task under either (A) goal-directed behavior (“model-based” reinforcement learning) or (B) habitual behavior (“model-free” or “cached” reinforcement learning). Reprinted by permission from Macmillan Publishers Ltd: Nature Neuroscience (Daw et al., 2005), copyright 2005.

Figure 6

A Bayesian interpretation of goal-directed learning proposed by Solway and Botvinick (2012). (A) Graphical model supporting the probabilistic factorization of a model-based reinforcement learning problem, and hence of goal-directed behavior, with a list of possible biological correspondents. ACC, anterior cingulate cortex; BA, Brodmann area; BLA, basolateral amygdala; dlPFC, dorsolateral prefrontal cortex; DLS, dorsolateral striatum; MF/PC, medial frontal/parietal cortex; MT, medial temporal cortex; PFC, prefrontal cortex; PC, parietal cortex; PMC, premotor cortex; SMA, supplementary motor area; vlPFC, ventrolateral prefrontal cortex. (B) A possible neural implementation of the functional architecture: based on (A) the reader might attempt to link neural areas to the components of the architecture. Adapted and reprinted with permission (Solway and Botvinick, 2012).

Figure 7

The proposal of Penner and Mizumori (2011) for the possible functions of nucleus accumbens core and shell, and their relation to downstream striatal regions. Notice the role of stimulus-outcome predictor ascribed to the accumbens. Reprinted from Penner and Mizumori (2011), Copyright 2011, with permission from Elsevier.

Figure 8

An hypothesis of ventral striatum as the locus of various types of action-outcome anticipations. Reprinted from Pennartz et al., (2011), Copyright 2011, with permission from Elsevier.

Figure 9

The three major systems for learning to select desired outcomes forming an evolutionary lineage used here as a background for our hypothesis. (A) First system formed by instrumental stimulus-response behaviors and simple Pavlovian processes. (B) Second system formed by instrumental stimulus-response behaviors and sophisticated Pavlovian processes supported by dynamical neural processes capable of sustained active representations of outcomes. (C) Third system formed by instrumental stimulus-response behaviors, sophisticated Pavlovian processes, and further structures allowing outcome representations to recall actions.

Figure 10

Sketch of the main functional elements of the hypothesis, with their possible biological correspondents. Amg, amygdala; DLS, dorsolateral striatum; DMS, dorsomedial striatum; Hip, hippocampus; ITC, inferotemporal cortex; M1, primary motor cortex; NAcc, nucleus accumbens; PC, parietal cortex; PFC, prefrontal cortex; PMC, premotor cortex.

Figure 11

Functional differences between the basolateral amygdala (BLA) and the central nucleus of amygdala (CeA). CS, conditioned stimulus; US, unconditioned stimulus; UR, unconditioned response.

Figure 12

Various components of the Hippocampal system underlying novelty detection in Hip, and the consequent production of dopamine via indirect connections to the VTA. Reprinted from Lisman and Grace (2005), Copyright 2005, with permission from Elsevier.

Figure 13

Anatomical differences between the basal ganglia circuits involving nucleus accumbens core and shell. (A) Overall schema of the connections involving the whole nucleus accumbens. (B) Zoom on the connections involving the nucleus accumbens shell. (C) Zoom on the connections involving the nucleus accumbens core. Reprinted from Humphries and Prescott (2010), Copyright 2010, with permission from Elsevier.

Figure 14

Anatomy and connections of nucleus accumbens, basolateral amygdala, hippocampus, and prefrontal cortex. Reprinted from Voorn et al. (2004), Copyright 2004, with permission from Elsevier.