Neural Population Dynamics Underlying Expected Value Computation

Abstract

Computation of expected values (i.e., probability × magnitude) seems to be a dynamic integrative process performed by the brain for efficient economic behavior. However, neural dynamics underlying this computation is largely unknown. Using lottery tasks in monkeys (Macaca mulatta, male; Macaca fuscata, female), we examined (1) whether four core reward-related brain regions detect and integrate probability and magnitude cued by numerical symbols and (2) whether these brain regions have distinct dynamics in the integrative process. Extraction of the mechanistic structure of neural population signals demonstrated that expected value signals simultaneously arose in the central orbitofrontal cortex (cOFC; medial part of area 13) and ventral striatum (VS). Moreover, these signals were incredibly stable compared with weak and/or fluctuating signals in the dorsal striatum and medial OFC. Temporal dynamics of these stable expected value signals were unambiguously distinct: sharp and gradual signal evolutions in the cOFC and VS, respectively. These intimate dynamics suggest that the cOFC and VS compute the expected values with unique time constants, as distinct, partially overlapping processes.SIGNIFICANCE STATEMENT Our results differ from those of earlier studies suggesting that many reward-related regions in the brain signal probability and/or magnitude and provide a mechanistic structure for expected value computation employed in multiple neural populations. A central part of the orbitofrontal cortex (cOFC) and ventral striatum (VS) can simultaneously detect and integrate probability and magnitude into an expected value. Our empirical study on these neural population dynamics raises a possibility that the cOFC and VS cooperate on this computation with unique time constants as distinct, partially overlapping processes.

Keywords: computation; expected values; integration; monkey; neural population dynamics; rewards.

Figure 1.

Task, behavior, and basic firing properties of neurons. A, Sequence of events during the single-cue task. A single visual pie chart having green and blue pie segments was presented to the monkeys. B, Choice task. Two visually displayed pie charts were presented to the monkeys on the left and right sides of the center. After visual fixation of the reappeared in the central area of the target, the central fixation target disappeared, and monkeys chose either of the targets by fixating on it. A block of the choice trials was sometimes interleaved between the single-cue trial blocks. During the choice trials, neural activity was not recorded. C, Percentages of right target choice during the choice task plotted against the EVs of the left and right target options. Aggregated choice data were used. D, Pseudo-r² estimated in the three behavioral models: M1, number of pie segments; M2, probability and magnitude; M3: expected values. E, Percentage of right target choices estimated in each recording session (gray lines) plotted against the difference in expected values (right minus left). The choice data were segmented by seven conditions of the difference in the expected values, as follows: −1.0 to −0.5, −0.5 to −0.3, −0.3 to −0.1, −0.1 to 0.1, 0.1 to 0.3, 0.3 to 0.5, and 0.5 to 1.0. Black plots indicate the mean. F, Reaction time to choose a target option plotted against the difference in expected values (right minus left) as −1.0 to −0.5, −0.5 to −0.3, −0.3 to −0.1, −0.1 to 0.1, 0.1 to 0.3, 0.3 to 0.5, and 0.5 to 1.0. G, An illustration of neural recording areas based on sagittal MR images. Neurons were recorded from the mOFC (14O, orbital part of area 14) and cOFC (area 13 M) at the A31–A34 anterior–posterior (A–P) level. Neurons were also recorded from the DS and VS, respectively, at the A21–A27 level. White scale bar, 5 mm. H, Color map histograms of neuronal activities recorded from the four brain regions. Each horizontal line indicates neural activity aligned to cue onset averaged for all lottery conditions. Neuronal firing rates were normalized to the peak activity. I, Percentages of neurons showing an activity peak during cue presentation. J, Box plots of peak activity latency after cue presentation. K, Firing rates of peak activity observed during cue presentation. L, Box plots of half-peak width, indicating the phasic nature of activity changes. M, Box plots of baseline firing rates during the 1 s time period before the onset of the central fixation target. In J–M, asterisks indicate statistical significance among two neural populations using the Wilcoxon rank-sum test with Bonferroni correction for multiple comparisons [statistical significance: **p < 0.01, *p < 0.05, and §0.05 < p < 0.06 (close to significance), respectively].

Figure 2.

Expected value signals detected by conventional analyses. A, Example activity histogram of a DS neuron modulated by expected value during the single-cue task. The activity aligned to the cue onset is represented for three different levels of probability (0.1–0.3, 0.4–0.7, and 0.8–1.0) and magnitude (0.1–0.3, 0.4–0.7, and 0.8–1.0 ml) of rewards. Gray hatched time windows indicate the 1 s time window used to estimate the neural firing rates shown in B. The neural modulation pattern was defined as the expected value type based on all three analyses (linear regression, AIC-based model selection, and BIC-based model selection). Regression coefficients for probability and magnitude were 6.17 (p < 0.001) and 2.54 (p = 0.007), respectively. B, An activity plot of the DS neuron during the 1 s time window shown in A against the probability and magnitude of rewards. C, D, Same as A and B, but for a VS neuron defined as the expected value type based on all three analyses. Regression coefficients for probability and magnitude were 7.14 (p < 0.001) and 6.71 (p < 0.001), respectively. E, F, Same as A and B, but for a cOFC neuron defined as the expected value type based on all three analyses. Regression coefficients for probability and magnitude were 8.55 (p < 0.001) and 11.1 (p < 0.001), respectively. G, H, Same as A and B, but for an mOFC neuron. The neural modulation pattern was defined as the expected value type based on the AIC-based model selection, as the probability type based on the linear regression, and as the nonmodulated type based on the BIC-based model selection. Regression coefficients for probability and magnitude were 1.76 (p = 0.032) and 0.50 (p = 0.54), respectively. I–L, Plots of regression coefficients for the probability and magnitude of rewards estimated for all neurons in the DS (I), VS (J), cOFC (K), and mOFC (L). Filled colors indicate the neural modulation pattern classified by the BIC-based model selection. P, Probability type; M, magnitude type, EV: Expected value type, and R-R: Risk-Return type. The nonmodulated type is indicated by the small open circle. M–P, Percentages of neural modulation types based on BIC-based model selection through cue presentation in the DS (M), VS (N), cOFC (O), and mOFC (P). The analysis window size is 0.1 s (left), 0.05 s (middle), and 0.02 s (right), respectively.

Figure 3.

Risk-return signals detected by conventional analyses. A, Example activity histogram of a VS neuron modulated by both probability and magnitude of rewards with opposite signs (i.e., negative b_p and positive b_m). The activity aligned to cue onset is represented for three different levels of probability (0.1–0.3, 0.4–0.7, and 0.8–1.0) and magnitude (0.1–0.3, 0.4–0.7, and 0.8–1.0 ml) of rewards. Gray hatched areas indicate a 1 s time window to estimate the neural firing rates shown in B. The neural modulation pattern was defined as the risk–return type based on the linear regression and AIC-based model selection, and as the magnitude type based on the BIC-based model selection. Regression coefficients were −2.44 (p = 0.039) and 4.86 (p < 0.001) for probability and magnitude, respectively. B, Activity plots of the VS neuron during the 1 s time window shown in A against the probability and magnitude of rewards. C, D, Same as A and B, but for a cOFC neuron. The neural modulation type was defined as the risk–return type based on all three analyses. Regression coefficients for probability and magnitude were −6.65 (p < 0.001) and 3.82 (p < 0.001), respectively.

Figure 4.

Schematic depictions for the analysis of neural population dynamics using PCA. A, Time series of a neural population activity projected into a regression subspace composed of probability and magnitude. A series of eigenvectors was obtained by applying PCA once to each of the four neural populations. PC1 and PC2 indicate the first and second principal components, respectively. The number of eigenvectors obtained by PCA was 2.7 s divided by the analysis window size for the probability and magnitude: 27, 54, and 135 eigenvectors in a 0.1, 0.05, or 0.02 s time window, respectively. B, Examples of eigenvectors at time of ith analysis window for probability and magnitude, whose direction indicates a signal characteristic at the time represented on the population ensemble activity. EV, 45°, 225°; M, magnitude (90°, 270°); P, probability (0°,180°); R-R, 135°, 315°. C, Characteristics of the eigenvectors evaluated quantitatively. Angle, Vector angle from the horizontal axis taken from 0° to 360°. Size, Eigenvector length; deviation, difference between vectors.

Figure 5.

Neural populations provide stable expected value signals in the VS and cOFC. A, Cumulative variance explained by PCA in the four neural populations. Dashed line indicates percentages of variances explained by PC1 and PC2 in each neural population. B, Overlay plots of series of eigenvectors for PC1 and PC2 in the four neural populations. a.u., Arbitrary unit. C, Box plots of vector deviation from the mean vector estimated in each neural population for PC1 (left) and PC2 (right). D, Box plots of vector size estimated in each neural population for PC1 (left) and PC2 (right). E–H, Same as A–D, but for the PCA under the shuffled condition 1. See Materials and Methods for details. I–L, Same as A–D, but for the PCA under the shuffled condition 2. In C, D, G, H, K, and L, asterisks indicate statistical significance between two populations using the Wilcoxon rank-sum test with Bonferroni correction for multiple comparisons [statistical significance at **p < 0.01, *p < 0.05, and §0.05 < p < 0.06 (close to significance), respectively]. The results are shown by using a 0.1 s analysis window.

Figure 6.

Probability density of explained variances by PCA in shuffled controls. A, Probability density of variances explained by PCA for PC1 to PC4 under the shuffled condition 1 (for details, see Materials and Methods). The probability density was estimated with 1000 repeats of the shuffle in each neural population. B, Probability density of variance explained by PCA for PC1 to PC4 under the shuffled condition 2 (for details, see Materials and Methods). The probability density was estimated with 1000 repeats of the shuffle in each neural population. In A and B, dashed lines indicate the variances explained by PCA in each of the four neural populations without the shuffle. The results are shown by using 0.1 s analysis window.

Figure 7.

Effects of the analysis window size on the PCA. A, Cumulative variances explained by PCA in the four neural populations. Dashed lines indicate the percentages of variance explained by PC1 and PC2 in each neural population. The sizes of the analysis window are 0.1, 0.05, and 0.02 s, respectively. B, Overlay plots of series of eigenvectors in the four neural populations. Eigenvectors for PC1 and PC2 are shown. The analysis window size is 0.1, 0.05, and 0.02 s, respectively. a.u., Arbitrary units. C, Box plots of vector deviation from the mean vector estimated in each neural population are shown for the PC1. D, Same as C, but for the PC2. E, Box plots of vector size estimated in each neural population are shown for the PC1. F, Same as E, but for the PC2. In C–F, asterisks indicate statistical significance between two neural populations using Wilcoxon rank-sum test with Bonferroni correction for multiple comparisons [statistical significance at **p < 0.01, *p < 0.05, and §0.05 < p < 0.06 (close to significance), respectively].

Figure 8.

Neural modulation patterns as regression coefficients in four neural populations. Plots of regression coefficients for the probability and magnitude of rewards estimated for all neurons in the DS, VS, cOFC, and mOFC. Regression coefficients when using a 0.1 s analysis window are shown every 0.5 s (0–0.1, 0.5–0.6, 1.0–1.1, 1.5–1.6, 2.0–2.1, and 2.5–2.6 s).

Figure 9.

Gradual and sharp evolutions of neural population signals in the VS and cOFC. A, Plots of eigenvector time series for PC1 in 0.02 s analysis windows shown in a sequential order during 1 s after cue onset. Horizontal and vertical scale bars indicate the eigenvectors for probability and magnitude in arbitrary units, respectively. B, Plots of the time series of vector size during 1 s after cue onset. Horizontal dashed lines indicate 3 SDs of the mean vector size during the baseline period, a 0.3 s time period before cue onset. Solid colored lines indicate interpolated lines using a cubic spline function to provide a resolution of 0.005 s. Vertical dashed lines indicate the onset (left) and peak (right) latencies for changes in vector sizes. C, Probability densities of onset latencies for the four neural population signals. Probability densities were estimated using bootstrap resamplings. Vertical dashed lines indicate means. Horizontal solid lines indicate bootstrap SEs. D, Same as C, but for peak latencies of the four neural population signals. E, Plots of time series of vector angle from the detected onset to the onset of outcome feedback. Solid black lines indicate regression slopes. In C and D, asterisks indicate statistical significance estimated using bootstrap resamplings (statistical significance at ***p < 0.001 and *p < 0.05, respectively). In E, triple asterisks indicate a statistical significance of the regression slope at p < 0.001. Data for PC2 are not shown.

Figure 10.

Neural population structures of the VS and cOFC with multiplicative integration of probability and magnitude. A, Cumulative variance explained by PCA in the four neural populations when the state space analysis was performed with the expected value into the regression matrix. Dashed line indicates the percentage of variances explained by PC1 and PC2 in each neural population. B, Plots of time series of eigenvectors connected with lines for PC1 to PC3 in the VS and cOFC. Eigenvectors during cue presentation were presented from the beginning to the end using a 0.1 s analysis window. Plots at the beginning and end are filled in black and labeled as start (s) and end (e), respectively. a.u., Arbitrary unit.