A cognitive map for value-guided choice in ventromedial prefrontal cortex

Abstract

The prefrontal cortex is crucial for economic decision-making and representing the value of options. However, how such representations facilitate flexible decisions remains unknown. We reframe economic decision-making in prefrontal cortex in line with representations of structure within the medial temporal lobe because such cognitive map representations are known to facilitate flexible behaviour. Specifically, we framed choice between different options as a navigation process in value space. Here we show that choices in a 2D value space defined by reward magnitude and probability were represented with a grid-like code, analogous to that found in spatial navigation. The grid-like code was present in ventromedial prefrontal cortex (vmPFC) local field potential theta frequency and the result replicated in an independent dataset. Neurons in vmPFC similarly contained a grid-like code, in addition to encoding the linear value of the chosen option. Importantly, both signals were modulated by theta frequency - occurring at theta troughs but on separate theta cycles. Furthermore, we found sharp-wave ripples - a key neural signature of planning and flexible behaviour - in vmPFC, which were modulated by accuracy and reward. These results demonstrate that multiple cognitive map-like computations are deployed in vmPFC during economic decision-making, suggesting a new framework for the implementation of choice in prefrontal cortex.

Figure 1:. Task design and canonical value representations.

a) Subjects were simultaneously presented with a left and right choice option, each comprising two cues (images). Once the cues were presented (Cue onset), subjects were free to saccade to the cues and choose their preferred option by moving the joystick towards it (Choice onset). The choice was confirmed by the highlighting of the selected option (Prefeedback), followed by reward delivery. The set of ten unique cues, each representing one of five reward magnitude or reward probability levels, was swapped every three to four recording sessions. Each stimulus set had ten new unique images, which represented the same set of five reward magnitude and reward probability levels (right panel). b) Accuracy (i.e., choosing the option with the highest expected value) across sessions. c) Psychometric curve across all trials. d) Chosen value coefficient of partial determination (CPD) for individual brain regions across the choice epoch. The thick line represents the mean response across the population within the brain region, the shaded area represents the standard error of the mean (SEM) across neurons within the brain region. Cue onset was approximately 560 msec before choice onset (mean reaction time across sessions). e) Mean CPD for the same regressor as in d) for individual brain regions within a 300-millisecond time window before subjects initiated their choice. The black line denotes the 95^th percentile of a null distribution for visual purposes, obtained through permutation testing (1000 permutations). *** p < .001. Error bars represent SEM across neurons within the brain region.

Figure 2:. The value space is represented with a grid-like code at choice in vmPFC.

a) Left panel: A value space is organised along the reward magnitude and reward probability values used for choice. Within this value space, the left and right options are embedded as “locations”. A trajectory or navigation angle can be computed between each pair of possible locations. Middle & right panel: Navigation angles falling along the grid field of a hypothetical grid cell (aligned) are predicted to elicit stronger oscillatory activity compared to angles that do not fall along the grid field (unaligned). b) Significant hexadirectional (sixfold) modulation in vmFPC but not control symmetries across recording sessions. Each session represents the average of several channels recorded within that session. See also Figure S3A. Error bars represent SEM across sessions (n = 16). * p_Bonferroni < .05, corrected for symmetries. c) Time course of hexadirectional (sixfold) modulation in vmPFC and other brain regions. Blue shading denotes the original time window in Figure 2B and Figure 1E. Lines above brain regions denote significant hexadirectional (sixfold) encoding at p < .05. See also Figure S3B. Error bars represent SEM across sessions within brain region (dlPFC = 32, OFC = 34, ACC = 41). d) Sixfold periodicity in vmPFC across sessions as predicted by Figure 2A, right panel. Error bars represent SEM across sessions. e) Percentage of significant sessions for individual symmetries obtained through permutation testing (n = 1000) by comparing session signal averages to the 99^th percentile of a null distribution. f) Sixfold periodicity for an example session. Error bars represent SEM across channels within that session (n = 3). g) Average signal for aligned compared to unaligned trials. Error bars represent SEM averaged across recorded channels (n = 3) within that session. h) Significant hexadirectional (sixfold) modulation in vmPFC but not control symmetries in an independent dataset occurring 100–500 msec after Option 2 onset. See Figure S5 for the task description. * p < .05. Error bars represent SEM across sessions (n = 11). i) Time course of hexadirectional (sixfold) modulation in vmPFC and another brain region (OFC) in an independent dataset. Error bars represent SEM across sessions within brain region (OFC, n = 8). j) Sixfold periodicity in vmPFC across sessions in an independent dataset. Error bars represent SEM across sessions.

Figure 3:. Grid orientations are stable within stimulus sets but realign across stimulus sets.

a) The stimulus set denoting reward magnitude and reward probability was swapped every few sessions. The stimuli denoting the five reward magnitude and reward probability levels were not repeated and did not correlate with one another across sessions. b) grid orientations are stable across session comparisons within stimulus sets as observed by hexadirectional (sixfold) modulation in vmPFC. ** p_Bonferroni < .01. Error bars represent SEM across session comparisons within stimulus sets (n = 9). c) grid orientation is not stable across session comparisons across stimulus sets. Error bars represent SEM across session comparisons across stimulus sets (n = 16). d) schematic representation of the observed effect – the orientation of a putative grid field with respect to the underlying value space rotates from one environment to another environment. e) The average difference in grid orientations from different sessions with shared stimulus sets was smaller than the average difference in grid orientation from different sessions with different stimulus sets. The black line denotes the empirical value obtained for the difference of between-within grid orientation distances. The gray histogram denotes the shuffled null distribution (n = 1000 permutations).

Figure 4:. vmPFC neurons are theta modulated and maintain a grid-like code and chosen value code in separate theta cycles.

a) Left panel: a sample theta cycle (blue) with its corresponding phase (black) showing the cycle–phase convention used throughout this figure. Right panel: vmPFC neurons were modulated by theta phase, firing most at theta troughs and firing least at theta peaks. Error bars represent SEM across vmPFC neurons. b) An example neuron plotted across all theta phases; representing how neurons responded on average (see S5BC for different patterns). c) The same example neuron from b) plotted at the Preferred and Non-Preferred phase – determined by t-tests across all cells (see Figure S7A). The neuron increased its firing closer to choice at a higher rate when indexed at theta troughs (Preferred) compared to theta peaks (Non-Preferred). Error bars represent SEM across trials within the session (n = 228). d) Average theta LFP across all vmPFC channels. The blue shading denotes the original time window in Figure 2B and Figure 1E. The numbers denote seed windows centred on theta throughs and peaks occurring within this time window. Error bars represent SEM across all vmPFC channels (n = 97). e) Significant hexadirectional (sixfold) modulation in vmPFC neurons at theta trough. * p_Bonferroni < .05. Error bars represent SEM across vmPFC neurons. f) Theta phase distribution across trials and time for an example channel before (left) and after (right) aligning it in time to one theta cycle. g) A grid-like code (green line), like in e), can be observed within one theta cycle before choice. The strength of the grid-like code is expressed as a t-value obtained from a t-test against zero across all cells using estimates from the hexadirectional analysis. Control symmetries are represented with the gray line. The dotted black lines represent t-test significance thresholds (significant t-test against zero) for visual purposes. The black bar denotes significance in cluster-based permutation testing (exceeding the 97.5^th percentile of the length-corrected null). h) Firing activity of an example neuron which exhibited a high grid-like code, plotted in the reward magnitude by reward probability space used to test for a grid-like code and averaged for the left option. The firing activity was averaged over the time period from Figure 4G where a significant grid-like code was found. i) firing activity of another example neuron with a high grid-like code, averaging over the left option. j) the same neuron from h) averaged for presentations of the right option. k) the same neuron from i) averaged for presentations of the right option. l) a chosen value signal can be observed at a different, subsequent, theta cycle before choice. The black bars denote significance in cluster-based permutation testing (exceeding the 97.5^th percentile of the length-corrected null). Error bar represents SEM across vmPFC neurons.

Figure 5:. Sharp wave ripples in the vmPFC.

a) Four simultaneously recorded channels. The top two rows show the wideband LFP signal where a candidate ripple was detected on both electrodes in vmPFC (purple shading). The third row from the top shows the top signal filtered to the ripple band (80 – 180Hz). The fourth row shows the smoothed envelope where the z-score crosses the threshold used to select candidate ripple events. The bottom two rows show the wideband LFP signal of two other channels recorded simultaneously at identical time points. Note that all simultaneously recorded LFP channels are from independent microelectrodes spaced a minimum of 1mm apart. b) Top panel: mean oscillatory activity across all ripple band events of all channels together. Bottom panel: average power spectrum of the above events superimposed across the same time period. c) The ripple events coincided with increased firing in vmPFC neurons. Error bar represents SEM across vmPFC neurons. d) Average high-pass filtered signal highlighting the sharp wave component.

Figure 6:. Accuracy and reward modulate ripple proportion during choice and rest.

a) Average probability of detecting ripples on a trial, averaged across all channels in all four brain regions. Error bars represent SEM across channels within each brain region (ACC = 149, dlPFC = 104, OFC = 133). b) Session-level accuracy can be predicted from ripple probability across channels in vmPFC before and after choice. The highlighted black lines represent clusters exceeding the 97.5^th percentile of a length-corrected null distribution. The gray shading represents the permutation thresholds of the null distribution. c) Correlation taken at peak of b) before choice, each dot represents a vmPFC channel. d) When subjects were previously rewarded, more ripples occur in the subsequent fixation period. Error bars represent SEM across vmPFC channels (n = 46). The black bar denotes significance in cluster-based permutation testing (exceeding the 97.5th percentile of the length-corrected null distribution).