Reward functions of the basal ganglia

Abstract

Besides their fundamental movement function evidenced by Parkinsonian deficits, the basal ganglia are involved in processing closely linked non-motor, cognitive and reward information. This review describes the reward functions of three brain structures that are major components of the basal ganglia or are closely associated with the basal ganglia, namely midbrain dopamine neurons, pedunculopontine nucleus, and striatum (caudate nucleus, putamen, nucleus accumbens). Rewards are involved in learning (positive reinforcement), approach behavior, economic choices and positive emotions. The response of dopamine neurons to rewards consists of an early detection component and a subsequent reward component that reflects a prediction error in economic utility, but is unrelated to movement. Dopamine activations to non-rewarded or aversive stimuli reflect physical impact, but not punishment. Neurons in pedunculopontine nucleus project their axons to dopamine neurons and process sensory stimuli, movements and rewards and reward-predicting stimuli without coding outright reward prediction errors. Neurons in striatum, besides their pronounced movement relationships, process rewards irrespective of sensory and motor aspects, integrate reward information into movement activity, code the reward value of individual actions, change their reward-related activity during learning, and code own reward in social situations depending on whose action produces the reward. These data demonstrate a variety of well-characterized reward processes in specific basal ganglia nuclei consistent with an important function in non-motor aspects of motivated behavior.

Keywords: Dopamine; Pedunculopontine nucleus; Striatum.

Fig. 1

Basic characteristics of phasic dopamine responses. a Two dopamine response components: initial detection response (blue), and subsequent value response (red) in a dot motion discrimination task. The motion coherence increasing from 0 to 50 % leads to better behavioral dot motion discrimination, which translates into increases of reward probability from p = 0.49 to p = 0.99 [dopamine neurons process reward probability as value (Fiorillo et al. 2003)]. The first response component is constant (blue), whereas the second component grows with reward value derived from probability (reward prediction error). From Nomoto et al. (2010). b Accurate value coding at the time of reward despite initial indiscriminate stimulus detection response. Blue and red zones indicate the initial detection response and the subsequent value response, respectively. After an unrewarded stimulus (CS-), surprising reward (R) elicits a positive prediction error response, suggesting that the prediction at reward time reflects the lack of value prediction by the CS-. From Waelti et al. (2001). c Inverse relationship of dopamine activations to aversiveness of bitter solutions. The activation to the aversive solution (black, Denatonium, a strong bitter substance) turns into a depression with increasing aversiveness due to negative value (red), suggesting that the activation reflects physical impact rather than punishment. Imp/s impulses per second, n number of dopamine neurons. Time = 0 indicates onset of liquid delivery. From Fiorillo et al. (2013b)

Fig. 2

Utility prediction error signal in monkey dopamine neurons. Red utility function derived from behavioral choices using risky gambles. Black corresponding, nonlinear increase of population response (n = 14 dopamine neurons) in same animal to unpredicted juice. Norm imp/s normalized impulses per second. From Stauffer et al. (2014)

Fig. 3

Reward processing in monkey pedunculopontine nucleus. a Phasic and sustained responses to reward-predicting stimuli. Imp/s impulses per second. From Kobayashi and Okada (2007). b Magnitude discriminating reward responses. Norm imp/s normalized impulses per second. From Okada et al. (2009)

Fig. 4

Reward processing in monkey striatum. a Pure reward signal in ventral striatum. The neuron discriminates between raspberry and blackcurrant juice irrespective of movement to left or right target, and irrespective of the visual image predicting the juice (top vs. bottom). Trials in rasters are ordered from top to bottom according to left and then right stimulus presentation. From Hassani et al. (2001). b Conjoint processing of reward (vs. no reward) and movement (vs. no movement) in caudate nucleus (delayed go-nogo-ungo task). The neuronal activities reflect the specific future reward together with the specific action required to obtain that reward. From Hollerman et al. (1998). c Action value coding of single striatal neuron. Activity increases with value (probability) for left action (left panel blue vs. orange), but is unaffected by value changes for right action (right panel), indicating left action value coding. Imp/s impulses per second. From Samejima et al. (2005). d Adaptation of reward expectation activity in ventral striatum during learning. In each learning episode, two new visual stimuli instruct a rewarded and an unrewarded arm movement, respectively, resulting in different reward expectations for the same movement. With rewarded movements (left), the animal’s hand returns quickly to the resting key after reward delivery (long vertical markers, right to reward). With pseudorandomly alternating unrewarded movements, the hand returns quickly after an unrewarded tone to the resting key in initial trials (top right), but subsequently returns before the tone (green arrows), indicating initial reward expectation that disappears with learning. The reward expectation-related neuronal activity (short dots) shows a similar development during learning (from top to bottom). From Tremblay et al. (1998)

Fig. 5

Social reward signals in monkey striatum. a Behavioral task. Two monkeys sit opposite each other across a horizontally mounted touch-sensitive computer monitor. The acting animal moves from a resting key towards a touch table to give reward to itself, the other animal, both, or none, depending on a specific cue on the monitor (not shown). b Neuronal activation when receiving own reward, either only for the actor (red) or for both animals (green), but no activation with reward only for the other (violet) or nobody (blue). c Activation to the cue predicting own reward (for actor only and for both, red and green), conditional on own action, and not occurring with conspecific’s action. d Activation following target touch predicting own reward, conditional on conspecific’s action (dotted lines), and not occurring with own action. b–d Imp/s impulses per second, from Báez-Mendoza et al. (2013)