Learning attentional templates for value-based decision-making

Abstract

Attention filters sensory inputs to enhance task-relevant information. It is guided by an "attentional template" that represents the stimulus features that are currently relevant. To understand how the brain learns and uses templates, we trained monkeys to perform a visual search task that required them to repeatedly learn new attentional templates. Neural recordings found that templates were represented across the prefrontal and parietal cortex in a structured manner, such that perceptually neighboring templates had similar neural representations. When the task changed, a new attentional template was learned by incrementally shifting the template toward rewarded features. Finally, we found that attentional templates transformed stimulus features into a common value representation that allowed the same decision-making mechanisms to deploy attention, regardless of the identity of the template. Altogether, our results provide insight into the neural mechanisms by which the brain learns to control attention and how attention can be flexibly deployed across tasks.

Keywords: attention; cognitive control; decision-making; parietal cortex; prefrontal cortex; reinforcement learning; reward learning; visual search.

Figure 1.. Attentional template learning task and behavior

(A) (Left) Example trial. Randomly colored target stimuli were presented at three locations (randomly chosen out of four possible locations). Monkeys made a saccade to a target to “choose” it and receive the associated reward. (Right) Reward amount (number of drops; radial axis) varied as a function of the angular distance in color space between the chosen color and the template. (B) Example trial sequence. After monkeys reached a learning criterion (see STAR Methods), the template color changed. This switch was uncued. (C) Reward magnitude (radial axis) associated with a stimulus color (angular axis) changes when template changes. (D) Sequence of templates for an example session. Template color was pseudo-randomly chosen (see STAR Methods). (E) Learning curves for monkey B (top, 69 blocks) and monkey S (bottom, 102 blocks). The probability to choose the current best target increased after the template switch (green), whereas the probability of choosing the target that would have been best for the previous template decreased (pink). Shaded areas represent SEM. Solid and dashed lines represent the monkey’s and model’s behavior, respectively. Gray dashed line indicates chance. (F) Mean absolute angular distance between the animal’s estimated template (the color with the highest expected value according to the model) and the true template color decreased with learning (mean ± SEM across blocks in 30 bins of normalized block length). (G) Examples of expected value distribution during learning. Brighter colors indicate higher expected value, according to the model. Previous and current templates are indicated with dashed lines. White marker indicates the peak of the expected value function on each trial. After a switch in template, the estimated template either (top) smoothly drifted toward the current template or (bottom) reset to uniform before reemerging at the new template. Resets (red line) were triggered by a large RPE, which occurred after a large change in the template color (Figures S1D and S1E). See also Figures S1 and S2.

Figure 2.. Distributed representation of the estimated attentional template in the parietal and frontal cortex

(A) Recording sites. (B) Example LIP and FEF neurons encoding template. Radial axis is firing rate (mean ± SEM, dashed lines) across trials in a −600- to 300-ms window around the onset of the targets. Angular axis is color of estimated template ($\text{bin}\text{size}=\pi /6$, smoothing of three bins). (C) Neuron responses in LPFC, projected onto the vector normal to the hyperplane for the three classifiers trained to discriminate pink, brown, and blue templates (same time window as B). Insets show schematic of decoder. Ellipses represent the mean (central dot) and the SEM (shaded) of the projection across 100 bootstraps. Color indicates the estimated template on withheld trials. (D) Classification accuracy of the estimated template in (top) LIP, (middle) FEF, and (bottom) LPFC, for each third of the block, computed on withheld trials. Multiclass classifier was either trained (left half) across all or (right half) within each progression level in the block. Violin plot: central white dot is the median, thick vertical gray bar represents the 25^th to 75^th quartile, and area represents the kernel density estimate of the data. **p ≤ 0.01, ***p ≤ 0.001. (E) Template representations are stable within a trial in (top) LIP (146 neurons), (middle) FEF (216 neurons), and (bottom) LPFC (475 neurons). Left: time course of response of neural population, projected into subspace of the second, third, and fourth principal component. Color indicates estimated template. Gray triangle indicates onset of targets. Green triangle indicates approximate time of response. (Right) classification accuracy of the estimated template in 300 ms windows either trained on the average activity in the full window (same as D and E) or trained for each window separately (100 bootstraps, mean and 95% confidence interval). Bar thickness above the data indicates significance level: thin, p ≤ 0.01 uncorrected; moderate, p ≤ 0.05 Bonferroni corrected across time (13 time points); and thick, p ≤ 0.01 Bonferroni corrected. See also Figure S3.

Figure 3.. Neural attentional templates are structured (A) Representation of estimated templates in neural population.

The population response to six template colors (independent bins, marker indicates color) into a two-dimensional space (defined by multidimensional scaling). LIP, 133 neurons; FEF, 216 neurons; LPFC, 459 neurons. (B) Schematic of decoding approach. Violin plot shows mean $Z$ scored circular distance between the neural and behavioral estimated template on withheld trials for each session. ***p ≤ 0.001. (C) Schematic of properties of a structured representation. (D) Circular mean neural and behavioral estimated template across blocks, color is that of the mean behavioral estimated template (r = 0.3015, p < 0.001, 171 templates). Thick dashed line indicates the circular correlation; as the line lies below the diagonal, this suggests a bias in templates, possibly due to biases in color perception and/or memory. (E) Histogram of the circular distances between mean neural and behavioral estimated templates across blocks (r = 0.1092, p < 0.001, 14,535 pairs). See also Figure S4.

Figure 4.. Neural attentional templates are incrementally updated in the structured space

(A) (Left, top) Schematic of predicted change in template toward chosen color following a positive reward prediction error (+RPE). (Left, bottom) Circular mean of behavioral and neural updates toward the chosen color across sessions (all on withheld trials). (Right) Same as left, but for −RPEs, where the estimated template moves away from the chosen color. (B) Circular mean of neural update toward the chosen color (± SEM) after +RPE as a function of the size of the behavioral update (865, 268, and 170 trials in each bin). Inset shows violin plot of mean $Z$ scored distance between the neural and the behavioral update toward the chosen color across validation trials for each session. (C) (Top row) mean (left) behavioral and (middle) neural update magnitude (± SEM) as a function of magnitude of +RPE (10 bins, smoothing of 4 bins) across all validation trials following +RPE. (Right) Violin plot of mean Pearson correlation between the update magnitude and the magnitude of +RPE across validation trials for each session. (Bottom row) same as top row, but for absolute distance between the estimated template before the update (i.e., at n-1) and the chosen color. For all panels, *p ≤ 0.05, **p ≤ 0.01, ***p ≤ 0.001. See also Figure S4.

Figure 5.. Re-mapping stimuli to support a generalized decision-making process across templates

(A) Schematic of two hypotheses for how the attentional template supports decisions: (1) animals learn template-specific decision boundaries or (2) templates transform sensory information into a generalized value signal, allowing for a fixed decision readout. (B) Classification accuracy (with 95% confidence interval) of the chosen value over time (median split within each session) for (top) LIP, (middle) FEF, and (bottom) LPFC. Computed on withheld trials from the same template color bin as the training trials (within, solid line) or in a different template color bin (across, dashed line). Chance level was 1/2 (gray dashed line). (C) Time course of the mean split-half reliability (with 95% confidence interval) of the chosen value (colored by area) and reward (green) regressors computed across all locations. Bars indicate significance of chosen value and reward, and their difference (black): thin, p ≤ 0.05; moderate, p ≤ 0.01; and thick, p ≤ 0.05; Bonferroni corrected (22 time points). (D) Same as (B), but for classifiers trained to decode the chosen color (balanced for the estimated template, 2 bins). (E) Same as (B), but for decoding choice at the top contralateral location. For (B, D, and E), bar thickness indicates significance level: thin, p ≤ 0.01; moderate, p ≤ 0.05 Bonferroni corrected; and thick, p ≤ 0.01 Bonferroni corrected. Bonferroni correction was across time (27/27/23 time points for B, ,D and E) and locations (4 locations for E). See also Figure S5.

Figure 6.. Value representations transformed from location-specific to global over time

(A) Time course of the mean split-half reliability (with 95% confidence interval) of the chosen value at contralateral locations (contra) and ipsilateral locations (ipsi, dashed line) for LPFC. Bars indicate significance of each hemifield (yellow) and their difference (black). (B) Same as (A), but for unchosen value. (C) (Top) Schematic of distance index. (Bottom) Violin plot: mean area under the curve of the representation reliability (distance index = 0) or alignment across locations across the whole time period (22 time points, Figure S5E) for LPFC. Distribution across bootstraps. *p ≤ 0.05, **p ≤ 0.01, ***p ≤ 0.001 all Bonferroni corrected (6 pairs). See Figure S6G or LIP, and FEF and Figure S6L for the “early” time window alignment. (D) Same as (A), but for the correlation between the contralateral chosen value representation (shown in A and Figure S5A) and the global chosen value representation (shown in Figure 5C) for LIP, FEF, and LPFC. Upper bars (alignment to global) indicate the significance of the correlation disattenuation between the two vectors. Lower bars (local reliability > alignment to global) indicate whether the correlation between the two vectors is significantly lower than the local chosen value reliability. (E) Summary of results and putative sequence of task operations. Colored rectangles reflect involvement of a brain region in an operation (LPFC: yellow; FEF: red; LIP: blue). For all panels, bar thickness indicates significance level: thin, p ≤ 0.01; moderate, p ≤ 0.05 Bonferroni corrected across time (22 time points); and thick, p ≤ 0.01 Bonferroni corrected. See also Figure S6.