Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making

Abstract

Considerable evidence has emerged to implicate ventromedial prefrontal cortex in encoding expectations of future reward during value-based decision making. However, the nature of the learned associations upon which such representations depend is much less clear. Here, we aimed to determine whether expected reward representations in this region could be driven by action-outcome associations, rather than being dependent on the associative value assigned to particular discriminative stimuli. Subjects were scanned with functional magnetic resonance imaging while performing 2 variants of a simple reward-related decision task. In one version, subjects made choices between 2 different physical motor responses in the absence of discriminative stimuli, whereas in the other version, subjects chose between 2 different stimuli that were randomly assigned to different responses on a trial-by-trial basis. Using an extension of a reinforcement learning algorithm, we found activity in ventromedial prefrontal cortex tracked expected future reward during the action-based task as well as during the stimulus-based task, indicating that value representations in this region can be driven by action-outcome associations. These findings suggest that ventromedial prefrontal cortex may play a role in encoding the value of chosen actions irrespective of whether those actions denote physical motor responses or more abstract decision options.

Figure 1.

Experimental Task. (A) Sequence and timing of events during stimulus-based reversal (top) and action-based reversal (bottom). An instruction signaled the trial type (choice or follow trial). The cue (presented for 1 s) signaled the beginning of the decision period of 1.5 s. Once the subject made their choice response, it was shown to them. The outcome (win or loss) was presented 6 s after the cue for 1.5 s followed by a jittered intertrial interval. (B) Contrasting the 2 experimental conditions. During action-based reversal learning, subjects were presented with a single nondiscriminating stimulus and they had to choose between 2 different actions (a slide or a press on a trackball device). During stimulus-based reversal learning, the subjects had to choose between 2 different discriminating stimuli that were presented randomly on both sides of the screen. The subject responded by pressing a button corresponding to the side of the screen that the chosen stimulus was presented.

Figure 2.

Behavioral data (single subject analysis). Trial-by-trial behavioral model fit of our fictitious update model for an example subject. The top row shows actual choices of the different decision options (sticks) and the model-predicted expected value for each option (lines). The colored bar on top show win trials in green and loss trials in red. We converted these choice probabilities into probabilities of switching and staying and plotted in the second row the actual switches (black stick) and the model-predicted switch probabilities (red lines). An independent logistic regression model was fitted to these switch data (small graphs to the right) to confirm that the switch probabilities of our fictitious update model significantly explained the actual switch data. The logistic fit is plotted in red, the data in blue circles with bigger diameters for overlaying data points.

Figure 3.

Behavioral data (group analysis). Switch probabilities as predicted by our computational model were grouped into 5 bins (x-axis) and were plotted against the actual switch (light gray) and stay probabilities (dark gray, y-axis) in both experimental conditions (action-based and stimulus-based reversal).

Figure 4.

Value-related activations in both experimental conditions. Left panels: results of a conjunction analysis testing for common effect for value-related signals in action-based and stimulus-based reversal. Significant effects were found in the ventromedial prefrontal cortex (vmpfc) and in the amygdalo–hippocampal junction (amyg/hippo) and corresponding blood oxygen level–dependent (BOLD) time course are plotted in the panels below. Right panels: results of a contrast testing for greater value-related effect during action-based than stimulus-based reversal. Significant effects were found in the supplementary motor area (SMA) and in the midcingulate cortex (midcing) with corresponding BOLD time courses plotted in the panels below.

Figure 5.

Activations correlating with prediction error in both experimental conditions. Top and middle panels: results of a conjunction analysis testing for common activation to the prediction error derived from our computational model in action-based and stimulus-based reversal. The blood oxygen level–dependent (BOLD) time course in the right panels reveal that trials with a high prediction error elicit strong evoked responses in ventral (vstr) and dorsal striatum (dstr) following the outcome. Bottom panels: results of a differential contrast comparing prediction error signals in action-based and stimulus-based reversal, revealing a region of supramarginal gyrus (smg) exhibiting stronger correlations with prediction error in the action compared to the stimulus-based task.

Figure 6.

Activations related to switching or staying in both experimental conditions. Left panels: results from a conjunction analysis testing for switch-related activations common to both action-based and stimulus-based reversal. Common effects were found in the dorsolateral prefrontal cortex (dlpfc), the lateral orbitofrontal cortex (ofc), and in the anterior insula (ins). Right panels: results from a conjunction analysis testing for stay vs switch responses in both action-based and stimulus-based reversal. Effects were found in the ventromedial prefrontal cortex (vmpfc) and in the posterior cingulate cortex (post cing) (error bar = SEM across subjects).

Figure 7.

Activation pattern related to stronger switch > stay response in the action-based compared with the stimulus-based task [interaction contrast: (AR switch > stay) – (SR switch > stay)]. Top panel. SPM showing a stronger switch > stay effect in the action-based reversal in the left intraparietal sulcus (ips). Bottom panel. Parameter estimates indicating the average activation to switch and stay trials in both task conditions (error bar = SEM across subjects).