Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis

Abstract

Reinforcement learning describes motivated behavior in terms of two abstract signals. The representation of discrepancies between expected and actual rewards/punishments-prediction error-is thought to update the expected value of actions and predictive stimuli. Electrophysiological and lesion studies have suggested that mesostriatal prediction error signals control behavior through synaptic modification of cortico-striato-thalamic networks. Signals in the ventromedial prefrontal and orbitofrontal cortex are implicated in representing expected value. To obtain unbiased maps of these representations in the human brain, we performed a meta-analysis of functional magnetic resonance imaging studies that had employed algorithmic reinforcement learning models across a variety of experimental paradigms. We found that the ventral striatum (medial and lateral) and midbrain/thalamus represented reward prediction errors, consistent with animal studies. Prediction error signals were also seen in the frontal operculum/insula, particularly for social rewards. In Pavlovian studies, striatal prediction error signals extended into the amygdala, whereas instrumental tasks engaged the caudate. Prediction error maps were sensitive to the model-fitting procedure (fixed or individually estimated) and to the extent of spatial smoothing. A correlate of expected value was found in a posterior region of the ventromedial prefrontal cortex, caudal and medial to the orbitofrontal regions identified in animal studies. These findings highlight a reproducible motif of reinforcement learning in the cortico-striatal loops and identify methodological dimensions that may influence the reproducibility of activation patterns across studies.

Figure 1

The temporal difference (TD) model describes a real-time course of reward prediction error (PE) signals; PEs transfer from the US to the CS as learning progresses. In contrast, trial-level models such as Rescorla-Wagner describe PE only at the US. Associative strength (conceptually close to value) signals build at the CS. It is easy to see the resemblance between TD error signal and the combination of PE and associative strength signals in trial-level models. * Before the asymptote is reached. At the asymptote, PE at the US disappears.

Figure 2

Pie charts show the percentage of studies in each condition that were included in producing the RPE ALE map.

Figure 3

Map of significant ALE clusters associated with the RPE contrast, with the activations in the striatum highlighted. Pie charts show the contribution of studies of a particular class to the bilateral striatum activation. Percentages are not corrected for base rate.

Figure 4

Map of significant ALE clusters associated with the RPE contrast, with the activations in the midbrain and frontal operculum highlighted. Pie charts show the contribution of studies of a particular class to each activation. Percentages are not corrected for base rate.

Figure 5

Conjunction map showing the overlap of ALE maps from individual subgroup analyses (Fixed, Individual, Pavlovian, Instrumental, Outcome PE, TD, Monetary, Liquid and Social), with the left putamen cluster (x=-22, y=6, z=9, cluster size = 30) from the conjunction analysis shown in green and marked with arrows.

Figure 6

Map of significant ALE clusters associated with the EV contrast. Pie charts show the contribution of studies of a particular class to the subgenual cingulate activation. Percentages are not corrected for base rate.