Heads for learning, tails for memory: reward, reinforcement and a role of dopamine in determining behavioral relevance across multiple timescales

Abstract

Dopamine has long been tightly associated with aspects of reinforcement learning and motivation in simple situations where there are a limited number of stimuli to guide behavior and constrained range of outcomes. In naturalistic situations, however, there are many potential cues and foraging strategies that could be adopted, and it is critical that animals determine what might be behaviorally relevant in such complex environments. This requires not only detecting discrepancies with what they have recently experienced, but also identifying similarities with past experiences stored in memory. Here, we review what role dopamine might play in determining how and when to learn about the world, and how to develop choice policies appropriate to the situation faced. We discuss evidence that dopamine is shaped by motivation and memory and in turn shapes reward-based memory formation. In particular, we suggest that hippocampal-striatal-dopamine networks may interact to determine how surprising the world is and to either inhibit or promote actions at time of behavioral uncertainty.

Keywords: dopamine; hippocampus; long term memory; nucleus accumbens; reinforcement learning; reward.

Figure 1

Cue-elicited dopamine release in NAc core during a two-option decision making task. (A) Left panel: Average (mean + s.e.m.) phasic dopamine release in NAc core elicited by cues signaling availability of either a known low (2 pellets) or a high reward (4 pellets) option for a fixed cost (16 lever presses) in rats who have either had little (<9 sessions) or extended (≥9 sessions) experience with these contingencies. Data taken from forced trials (where only one option was available) after animals were choosing the high reward option on ≥75% of choice trials. Contingencies changed every session so rats had to flexibly update associations in each session. Right panel: Difference in cue-evoked peak dopamine release between cues signaling high and low reward options as a function of number of sessions in which they had experienced these contingencies. There was no correlation between experience of dopamine-based benefit encoding. (B) Left panel: Same as (A) except now the benefit was fixed (2 pellets) and the cost varied across options (low effort = 2 lever presses; high effort = 16 lever presses). Right panel: Same as (A) for high and low effort costs. Now there was a significant reduction in dopamine cost encoding as a function of experience. n.s. differences not significant; ^*significant at p < 0.05. [adapted from Gan et al. (2010)].

Figure 2

Effects of natural and pharmacological reinforcers on the acquisition of cue- and spatially-guided learning strategies. (A, B) Left panels depict schematics of a cue-guided (A) and a spatially guided (B) version of a Y-maze decision making task. Correct responses were guided either by intramaze visual cues or by spatial location, respectively, and were reinforced in separate groups of mice by either food reward or intra-VTA morphine infusions. Control mice received intra-VTA aCSF infusions and no food reward at the “correct” location. Middle panels depict choice performance on the 10th day of training on the respective task (chance performance = 50%, marked with dashed line). Right panels depict pCREB levels measured in NAc, dorsal striatum (DS), and the hippocampus (HPC) of food- and morphine-reinforced mice after 10 days of training on the cued (A: upper) or the spatial (B: middle) version of the Y-maze task, normalized to pCREB levels observed in aCSF controls. (C) Relative changes in pCREB levels in HPC, NAc, and DS after training on the spatial as compared to the cued task (cued task pCREB = 100%, dashed line). (D) Left: Effect of daily intra-DS injection of either the PKA inhibitor Rp-cAMPS or aCSF on the spatial version of the Y maze in morphine reinforced mice. Left: Choice performance on the 10th training day. Right: pCREB levels measured in NAc and HPC after 10 days of training on the spatial task in morphine-reinforced mice that received intra-DS injections of Rp-cAMPS, normalized to morphine-reinforced aCSF controls (dashed line). n.s. differences not significant, ^*significant at p < 0.05. [adapted from Baudonnat et al. (2011)].