Working memory management and predicted utility

Abstract

Given the limited capacity of working memory (WM), its resources should be allocated strategically. One strategy is filtering, whereby access to WM is granted preferentially to items with the greatest utility. However, reallocation of WM resources might be required if the utility of maintained information subsequently declines. Here, we present behavioral, computational, and neuroimaging evidence that human participants track changes in the predicted utility of information in WM. First, participants demonstrated behavioral costs when the utility of items already maintained in WM declined and resources should be reallocated. An adapted Q-learning model indicated that these costs scaled with the historical utility of individual items. Finally, model-based neuroimaging demonstrated that frontal cortex tracked the utility of items to be maintained in WM, whereas ventral striatum tracked changes in the utility of items maintained in WM to the degree that these items are no longer useful. Our findings suggest that frontostriatal mechanisms track the utility of information in WM, and that these dynamics may predict delays in the removal of information from WM.

Keywords: Q-learning; filtering; gating; predicted utility; working memory.

Figure 1

Task rules (A), theinterstimulus interval manipulation (B), and control conditions (C). (A) In this sequential WM task, participants are instructed that digits serve as context in determining which class of lower-level items will be relevant for selecting a response. For example, if the digit “1”/“2” appears, wingdings/letters are relevant for responses (respectively), whereas if the digit “3” appears both of the lower-level items will be relevant. Participants are to identify the relevant centrally-presented item(s) from each trial at the bottom of the final stimulus display, and press a corresponding key. (B) In the ICR condition, an irrelevant item precedes the context. With a “short” ISI, this irrelevant item may increase the effective WM load; following a longer ISI, the influence of this irrelevant item may be reduced. (C) In the RCI control condition, the relevant lower-level item is presented prior to the digit, so participants must merely identify the location of that relevant stimulus at the bottom of the final stimulus display; by contrast, in the RCR control condition, participants may experience a higher effective WM load because both items are relevant for a response.

Figure 2

Disproportionate benefits as a function of ISI in the ICR condition in terms of errors (A and B) and reaction times (C and D). (A) Error rates decreased as a function of ISI in the ICR condition. (B) The majority of the data fell reliably below the line of identity relating error rates on the short and long ISI trials in the ICR condition, indicating a reliable increase in accuracy as a function of ISI in that condition. [Note: participants with perfect accuracy across both ISIs are not shown (n = 6)]. (C) As with Errors, RT decreased as a function of ISI in the ICR condition, relative to the control conditions. (D) Again, the majority of the data fell below the line of identity relating RT on the short and long ISI trials in the ICR condition, indicating a reliable decrease in RT as a function of ISI.

Figure 3

Conceptual schematic of computational model (A), trial-by-trial RL-like updating of utilities (B), and significant fits to RT in the ICR condition as a function of ISI (C). (A) The key conceptual assumption in our model is that the benefit of a longer ISI to RT and accuracy in the ICR condition reflects a sluggish reduction in effective WM load following the presentation of context, but that this sluggishness is exacerbated for irrelevant items that had a higher predicted utility. (B) Several examples illustrate how predicted and experienced utilities for an example item (“”) change across trials. In the first illustrated trial, is specified as relevant by the context and is associated with the correct response, therefore acquiring an experienced utility of 1; the predicted utility on the following trial is updated by the resulting utility prediction error, scaled by the learning rate (here, 0.09). On this second trial, “” is specified as irrelevant (due to the context) but is still associated with the correct response, and thus again acquires an experienced utility of 1. This experience changes its predicted utility for the third trial, where it is now associated with the incorrect response, and acquires zero experienced utility. In the fourth trial, is not eligible for experienced utility because it should not be held within WM (the context had rendered it irrelevant), whereas in the fifth and sixth trials, R is associated/not associated (respectively) with the correct response and acquires utilities of 1/0, respectively. (C) At the short ISI of the ICR condition, the predicted utility of irrelevant items was positively correlated with RT (“irrelevant utility cost”), whereas the predicted utility of the relevant items was negatively correlated (“relevant utility benefit”). These effects were diminished at the long ISI.

Figure 4

BOLD responses to context vs. fixation (red-yellow) and common areas of recruitment (black outlines). Relative to fixation, each of the three conditions elicited a reliable BOLD response across frontal, parietal, and occipital cortex. Most significant frontal activation, whether medial and lateral, was observed in the left hemisphere (voxelwise z > 2.3, corrected to p < 0.05 via GRF).

Figure 5

BOLD responses to selective vs. global context conditions (red-yellow) and common areas of recruitment (black outlines). Contrasts of RCI and ICR vs. RCR revealed reliable BOLD response across frontal, parietal, and occipital cortex (voxelwise z > 2.3, corrected to p < 0.05 via GRF). The reverse contrasts of RCR > ICR and RCR > RCI, and the direct contrasts of ICR with RCI, failed to reach significance.

Figure 6

BOLD correlates of model-based Predicted Utility estimates. In the RCI condition, trials associated with lower predicted utility at the presentation of context elicited a reliably stronger BOLD response in visual cortex, anterior intraparietal sulcus, and a region of dorsal premotor cortex, both relative to fixation (A) and relative to the ICR condition (B). A similar effect was observed in scattered sections of the prefrontal cortex for the RCR condition, including left rostrolateral prefrontal cortex, left dorsal premotor cortex, and ventral prefrontal cortex (C), although only activation in the occipital lobe differentiated these effects of RCR from the ICR condition (D). Voxelwise z > 2.3, corrected to p < 0.05 via GRF.

Figure 7

Ventral striatal BOLD positively and differentially correlates with the predicted utility of items in WM in the ICR condition. (A) Relative to all other conditions, bilateral ventral striatum was more strongly correlated with the predicted utility of items in WM in the ICR condition. (B) After restricting our analyzed volume to the basal ganglia (all other regions masked), left ventral striatum was found to show a positive correlation with the predicted utility of items in WM in the ICR condition. No other condition showed this effect, and indeed bilateral ventral striatum was more positively correlated with predicted utility in the ICR condition than either the RCI (C) or RCR (D) conditions alone. Note that (B–D) mask out activity outside of the basal ganglia (including caudate, nucleus accumbens, pallidum, and putamen). Voxelwise z > 2.3, corrected to p < 0.05 via GRF.