Dopamine in motivational control: rewarding, aversive, and alerting

Abstract

Midbrain dopamine neurons are well known for their strong responses to rewards and their critical role in positive motivation. It has become increasingly clear, however, that dopamine neurons also transmit signals related to salient but nonrewarding experiences such as aversive and alerting events. Here we review recent advances in understanding the reward and nonreward functions of dopamine. Based on this data, we propose that dopamine neurons come in multiple types that are connected with distinct brain networks and have distinct roles in motivational control. Some dopamine neurons encode motivational value, supporting brain networks for seeking, evaluation, and value learning. Others encode motivational salience, supporting brain networks for orienting, cognition, and general motivation. Both types of dopamine neurons are augmented by an alerting signal involved in rapid detection of potentially important sensory cues. We hypothesize that these dopaminergic pathways for value, salience, and alerting cooperate to support adaptive behavior.

Figure 1. Dopamine coding of reward prediction errors and preference for predictive information

(A) Conventional theories of DA reward signals. DA neurons encode a reward prediction error signal, responding with phasic excitation when a situation’s reward value becomes better than predicted (red) and phasic inhibition when the value becomes worse than expected (blue). These signals could be used for learning, to reinforce or punish previous actions (backward arrows) or for immediate control of behavior, to promote or suppress reward-seeking actions (forward arrows). (B–E) An example DA neuron with conventional coding of reward prediction errors as well as coding of the subjective preference for predictive information. Each plot shows the neuron’s mean firing rate (histogram, top) and its spikes on 20 individual trials (bottom rasters) during each condition of the task. Data is from (Bromberg-Martin and Hikosaka, 2009). (B) This DA neuron was excited by a cue indicating that an informative cue would appear to tell the size of a future reward (red). (C) DA excitation by a big reward cue (red), inhibition by a small reward cue (blue), and no response to predictable reward outcomes (black). (D) This DA neuron was inhibited by a cue indicating that an uninformative cue would appear which would leave the reward size unpredictable (blue). (E) DA lack of response to uninformative cues (black), excitation by an unexpectedly big reward (red), and inhibition by an unexpectedly small reward (blue).

Figure 2. Dopamine control of positive and negative motivation in the dorsal striatum

(A) If an action is followed by a new situation that is better than predicted, DA neurons fire a burst of spikes. This is thought to activate D1 receptors on direct pathway neurons, promoting immediate action as well as reinforcing cortico-striatal synapses to promote selection of that action in the future. (B) If an action is followed by a new situation that is worse than predicted, DA neurons pause their spiking activity. This is thought to inhibit D2 receptors on indirect pathway neurons, promoting suppression of immediate action as well as reinforcing cortico-striatal synapses to promote suppression of that action in the future.

Figure 3. Diverse dopamine neuron responses to aversive events

Two example DA neurons in the VTA that were phasically inhibited (top) or excited (bottom) by noxious footshocks. These neurons were recorded in anesthetized rats and were confirmed to be dopaminergic using juxtacellular labeling. Adapted from (Brischoux et al., 2009).

Figure 4. Distinct dopamine neuron populations encoding motivational value and salience

(A) Motivational value coding DA neurons are excited by reward cues and reward outcomes (fruit juice) and inhibited by aversive cues and aversive outcomes (airpuffs). (B) Motivational salience coding DA neurons are excited by both reward and aversive cues and outcomes. Analysis and classification of neurons adapted from (Bromberg-Martin et al., 2010a); original data from (Matsumoto and Hikosaka, 2009b).

Figure 5. Hypothesized functions of motivational value, salience, and alerting signals

Hypothesized functions of motivational signals in DA neurons. Motivational value signals are sent to value coding DA neurons which instruct seeking of rewards, evaluation of outcomes, and value learning. Motivational salience signals are sent to salience coding DA neurons which support attentional orienting, cognitive processing, and general motivation. Alerting signals are sent to both populations. In value coding DA neurons they promote seeking of environments where alert cues are available so that salient outcomes can be anticipated in advance. In salience coding DA neurons they implement this anticipation by promoting orienting to alert cues and deployment of cognitive and motivational resources.

Figure 6. Dopamine neuron excitatory responses to alerting events

DA neurons are excited by sensory cues that are alerting (left, black) but do not respond when the same cues are rendered non-alerting (right, gray). (A) A DA neuron bursts in response to an unexpected 113 dB auditory click. These bursts occur when the cat is in a state of quiet waking, but not when the cat is preoccupied by the presence of inaccessible food. Adapted from (Strecker and Jacobs, 1985). (B) A DA neuron bursts in response to a new sensory stimulus (a door opening). These bursts occur when it is relatively novel (presentations 12–33) but not when it is familiar (presentations 56–75). Histograms built using data from a neuron reported in (Ljungberg et al., 1992). (C,D) DA neurons that are excited by unexpected visual cues during tasks when the cues are potentially rewarding or aversive. In separate blocks of trials, animals were presented with cues and outcomes that were potentially rewarding (top, reward task) or aversive (bottom, aversive task). Data is the averaged activity of four motivational salience coding DA neurons; for clarity, stimulus colors have been modified and only a subset of conditions are shown. Adapted from (Bromberg-Martin et al., 2010a). (C) In a first experiment, motivational cues are presented with unpredictable timing. The DA neurons are excited by all cues, even a neutral cue (black curve) that had never been paired with rewarding or aversive outcomes. (D) In a second experiment, the timing of motivational cues is made fully predictable by presenting a “trial start cue” one second in advance. The DA neurons are no longer excited by the neutral cue (right, gray); instead, their excitation shifts to the trial start cue (left, black).

Figure 7. Hypothesized anatomical location and projections of dopamine motivational value and salience coding neurons

(A) In our hypothesis, motivational salience coding DA neurons are located predominantly in the dorsolateral SNc and medial VTA. They may send signals to regions of the nucleus accumbens core (NAc core), dorsal striatum, and dorsal and lateral prefrontal cortex (DLPFC). Motivational value coding DA neurons are located predominantly in the ventromedial SNc and throughout the VTA. They may send signals to regions of the nucleus accumbens shell (NAc shell), dorsal striatum, and ventromedial prefrontal cortex (VMPFC). (B) DA excitatory responses to aversive cues (red dots) often occur in the dorsolateral SNc, while inhibitory responses (blue dots) often occur in the ventromedial SNc. Data are from one monkey and collapsed across three adjacent 1 mm sections. Also labeled are the substantia nigra pars reticulata (SNr) and red nucleus (RN). Adapted from (Matsumoto and Hikosaka, 2009b). (C) DA neurons with greater excitation (red dots) or inhibition (blue dots) to aversive cues than neutral cues are mixed within the medial VTA. Also shown are neurons that had greater responses to neutral cues than aversive cues (gray dots). Also labeled is the fasciculus retroflexus (fr). Data are from eight rabbits and collapsed across three adjacent sections. Adapted from (Guarraci and Kapp, 1999).

Figure 8. Hypothesized sources of motivational value, salience, and alerting signals

In our hypothesis, motivational salience signals are sent to DA neurons through the central amygdala (CeA). Motivational value and alerting signals may be sent to DA neurons through a pathway including the globus pallidus border (GPb), lateral habenula (LHb), and rostromedial tegmental nucleus (RMTg). Value signals related to aversive outcomes may also be sent by the parabrachial nucleus (PBN), while alerting signals may also be sent by the superior colliculus (SC) and pedunculopontine tegmental nucleus (PPTg).