The neural code of reward anticipation in human orbitofrontal cortex

Abstract

An optimal choice among alternative behavioral options requires precise anticipatory representations of their possible outcomes. A fundamental question is how such anticipated outcomes are represented in the brain. Reward coding at the level of single cells in the orbitofrontal cortex (OFC) follows a more heterogeneous coding scheme than suggested by studies using functional MRI (fMRI) in humans. Using a combination of multivariate pattern classification and fMRI we show that the reward value of sensory cues can be decoded from distributed fMRI patterns in the OFC. This distributed representation is compatible with previous reports from animal electrophysiology that show that reward is encoded by different neural populations with opposing coding schemes. Importantly, the fMRI patterns representing specific values during anticipation are similar to those that emerge during the receipt of reward. Furthermore, we show that the degree of this coding similarity is related to subjects' ability to use value information to guide behavior. These findings narrow the gap between reward coding in humans and animals and corroborate the notion that value representations in OFC are independent of whether reward is anticipated or actually received.

Fig. 1.

Experimental design and behavioral results. (A) After presentation of a sensory cue, subjects had to judge either the main rotation direction or the color of the dots (randomized). The reward outcome was delivered after correct responses. (B) Sensory cues consisted of colored rotating dots that were associated with reward value in a logical XOR fashion. The combination of rotation direction and color was reward predicting, whereas color and rotation direction alone were not informative about the outcome. Two stimulus combinations that do not share any sensory properties predict high rewards (e.g., G&CW and R&CCW) and two predict low rewards (e.g., G&CCW and R&CW). CW, clockwise; CCW, counterclockwise. An example pairing is shown here (the actual parings were counterbalanced across subjects). All 16 cells were presented once in each run. To compensate for the overrepresentation of intermediate values, the four extreme values were presented two additional times each (16 + 2 × 4 = 24 trials). (C) Subjective ratings from a postscanning rating session increased as a function of reward value, suggesting that subjects indeed were aware of the link between cues and reward levels. All differences are significant (P < 0.001, Bonferroni corrected). Error bars for SEM are smaller than the symbols.

Fig. 2.

Multivariate pattern classification. (A) Cues were sorted into four groups according to their sensory properties. Two cues predicted a high reward (7 and 10 points), whereas the remaining two predicted a low reward (0 and 3 points). (B) A support vector classifier (SVC) was trained on "training" data from nine scanning runs to classify fMRI patterns evoked by one specific pair of low- vs. high-value cues (e.g., G&CCW vs. R&CCW). From the remaining test data set (run 10), fMRI patterns to cues that also predicted low and high values but had different sensory properties were used to test the performance of the SVC (e.g., R&CW vs. G&CW). In total, this procedure was performed on four different training-test pairs each time as a 10-fold leave-one-out cross-validation. (C) We searched in every local cluster of brain activity for information about the reward value during anticipation using a searchlight approach (37, 38). For every voxel in the brain, the fMRI patterns in the local cluster surrounding this voxel were extracted for each cue and each scanning run separately. Then the decoding procedure described in B was performed on that data.

Fig. 3.

Decoding of reward value during anticipation. (A) Distributed fMRI patterns in the medial OFC [MNI coordinates: (3, 33, −6), t = 6.90] and the ventral striatum [VS (6, 6, −6), t = 5.65] represent the value of anticipated outcomes independent of the sensory properties of the cues. T map based on the decoding accuracies of all four training-test pairs is thresholded at P < 0.05, FWE whole-brain corrected with a cluster extent threshold of k = 30 voxels, and overlaid on a normalized T1-weighted image averaged across subjects. (B) Bar graphs show average decoding accuracy across subjects (% correct classified, chance level is 50%) for the different training-test pairs (nos. 1–4; see Fig. 2B) and error bars depict SEM. Please note that the decoding accuracy only provides a lower bound on information. The predictive accuracy at the level of populations of single cells could potentially be substantially higher if only a subpopulation of cells is modulated by reward, as suggested by electrophysiological studies in primates (10, 24, 26).

Fig. 4.

Similar value-coding fMRI patterns in the OFC during anticipation and receipt of reward. (A) In the medial OFC [MNI coordinates: (3, 54, −15), t = 6.20] similar fMRI patterns represent value during both anticipation and receipt of reward. The t map based on decoding accuracies from both training-test pairs is thresholded at P < 0.05 (FWE whole-brain corrected; cluster extent threshold k = 30 voxels) and overlaid on a normalized T1-weighted image averaged across subjects. (B) The surface plot depicts voxel selectivities (support vector weights, SV weights) in the spherical cluster surrounding the individual peak voxel in medial OFC for one subject. The selectivity of each voxel for either low or high values is color coded in blue and yellow, respectively. (Left) SV weights from the SVC trained on fMRI patterns during anticipation and (Right) during receipt of reward. Scatter plot in the middle illustrates the similarity between the voxel selectivities during anticipation (x axis) and receipt of reward (y axis). 3D patterns from all subjects are shown in Figs. S3 and S4. (C) Significant relationship (r = 0.51, P < 0.05) between the correlation of the fMRI patterns in the medial OFC during anticipation and receipt of reward (pattern similarity, x axis) and the coefficients of determination (R²) describing the subjective association between the sensory cue and reward value (subjective association, y axis) obtained from the postscanning ratings. There was no significant relationship in the mPFC (P = 0.65) or the dACC cluster (P = 0.09). (D) Significant relationship (r = 0.61, P < 0.05) between pattern similarity in OFC (x axis) and the modulation of performance (% correct) by expected value (y axis) during the task. There was no significant relationship in the mPFC (P = 0.14) or the dACC (P = 0.29).