Separate neural representations of prediction error valence and surprise: Evidence from an fMRI meta-analysis

Abstract

Learning occurs when an outcome differs from expectations, generating a reward prediction error signal (RPE). The RPE signal has been hypothesized to simultaneously embody the valence of an outcome (better or worse than expected) and its surprise (how far from expectations). Nonetheless, growing evidence suggests that separate representations of the two RPE components exist in the human brain. Meta-analyses provide an opportunity to test this hypothesis and directly probe the extent to which the valence and surprise of the error signal are encoded in separate or overlapping networks. We carried out several meta-analyses on a large set of fMRI studies investigating the neural basis of RPE, locked at decision outcome. We identified two valence learning systems by pooling studies searching for differential neural activity in response to categorical positive-versus-negative outcomes. The first valence network (negative > positive) involved areas regulating alertness and switching behaviours such as the midcingulate cortex, the thalamus and the dorsolateral prefrontal cortex whereas the second valence network (positive > negative) encompassed regions of the human reward circuitry such as the ventral striatum and the ventromedial prefrontal cortex. We also found evidence of a largely distinct surprise-encoding network including the anterior cingulate cortex, anterior insula and dorsal striatum. Together with recent animal and electrophysiological evidence this meta-analysis points to a sequential and distributed encoding of different components of the RPE signal, with potentially distinct functional roles.

Keywords: decision making; learning; meta-analysis; prediction error; reward.

Figure 1

Hypothesized profiles for BOLD responses as function of the three RPE components. Pattern A (i and ii) describe the two categorical valence responses (yellow and blue colours indicate (i) responses being greater for negative compared with positive outcomes [NEG > POS] and (ii) responses being greater for positive compared with negative outcomes [POS > NEG]). Pattern B captures surprise effects with greater responses for higher outcome deviations from expectations, independent of the sign (valence) of the RPE. Pattern C shows a monotonically increasing response profile consistent with a signed RPE representation [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 2

Results of whole‐brain ALE analysis along the valence component. Overlays of brain areas activated by correlations with NEG > POS (blue) and POS > NEG (orange) (Pattern A [i] and [ii], respectively; Figure 1) (p values corrected with FDR‐ID [FID] and FDR‐pN [FRN] < .05 and a minimum cluster volume of 50 mm³). Representative slices are shown with MNI coordinates given below each image [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 3

Results of the whole brain ALE analysis for the surprise component of RPE (pattern B, Figure 1). Overlay of brain areas activated by all analyses representing direct or indirect measures of the surprise component of RPE (p values corrected with FDR‐ID [FID] and FDR‐pN [FRN] < .05 and a minimum cluster volume of 50 mm³). Representative slices are shown with MNI coordinates given below each image [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 4

Results of the ALE conjunction analysis between valence and surprise (purple). The regions identified earlier with separate ALE analyses along the valence (NEG > POS: blue, POS > NEG: orange) and surprise (green) components are shown for comparison purposes. The p values were corrected with FDR‐pN [FRN] < .05 and a minimum cluster volume of 50 mm³ for the initial maps. Representative slices are shown with MNI coordinates given bellow each image [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 5

Results of the ALE contrast analyses for [valence – surprise] (left panel) and [surprise – valence]. The p values were corrected with FDR‐pN [FRN] < .05 and a minimum cluster volume of 50 mm³ for the initial maps. Representative slices are shown with MNI coordinates given bellow each image [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 6

Results of whole brain ALE analysis for signed RPE. Overlay of brain areas activated by positive correlation with signed RPE (p values corrected with FDR‐ID [FID] and FDR‐pN [FRN] < .05 and a minimum cluster volume of 50 mm³). Representative slices are shown with MNI coordinates given bellow each image [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 7

Results of the ALE conjunction analysis for all components of RPE. Overlay of brain areas individually activated by (1) valence (orange), (2) surprise (green), and (3) signed RPE (red), with p values corrected with FDR‐pN [FRN] < .05 and a minimum cluster volume of 50 mm³ for the initial maps. Importantly, the overlap between the three analyses, shown in white, also corresponds to the only cluster found for the ALE conjunction analysis between valence/surprise vs. signed RPE. MNI coordinates are given below each image [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 8

Results of the ALE contrast analyses for [signed RPE – positive valence] (left panel), [signed RPE – negative valence] (middle panel) and [signed RPE – (positive + negative valence)] (right panel). The p values were corrected with FDR‐pN [FRN] < .05 and a minimum cluster volume of 50 mm³ for the initial maps. Representative slices are shown with MNI coordinates given bellow each image [Color figure can be viewed at http://wileyonlinelibrary.com]