Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis

Abstract

A number of recent advances have been achieved in the study of midbrain dopaminergic neurons. Understanding these advances and how they relate to one another requires a deep understanding of the computational models that serve as an explanatory framework and guide ongoing experimental inquiry. This intertwining of theory and experiment now suggests very clearly that the phasic activity of the midbrain dopamine neurons provides a global mechanism for synaptic modification. These synaptic modifications, in turn, provide the mechanistic underpinning for a specific class of reinforcement learning mechanisms that now seem to underlie much of human and animal behavior. This review describes both the critical empirical findings that are at the root of this conclusion and the fantastic theoretical advances from which this conclusion is drawn.

Fig. 1.

“Weights determining the effects of previous rewards on current associative strength effectively decline as an exponential function of time” (65). [Reproduced with permission from Oxford University Press from ref. (Copyright 2010, Paul W. Glimcher).]

Fig. 2.

“Raster plot of dopamine neuron activity. Upper panel shows response of dopamine neuron to reward before and after training. Lower panel shows response of dopamine neuron to start cue after training” (26). [Reproduced with permission from ref. (Copyright 1993, Society for Neuroscience).]

Fig. 3.

“When a reward is cued and delivered, dopamine neurons respond only to the cue. When an expected reward is omitted after a cue the neuron responds with a suppression of activity as indicated by the oval” (29). [Reproduced with permission from ref. (Copyright 1997, American Association for the Advancement of Science).]

Fig. 4.

“The linear weighting function which best relates dopamine activity to reward history” (65). [Reproduced with permission from Oxford University Press from ref. (Copyright 2011, Paul W. Glimcher).]

Fig. 5.

“Peri-stimulus time histogram of dopamine neuron activity during a cued and probabilistically rewarded task” (37). [Reproduced with permission from ref. (Copyright 2003, American Association for the Advancement of Science).]