Manipulating attentional priority creates a trade-off between memory and sensory representations in human visual cortex

Abstract

People often remember visual information over brief delays while actively engaging with ongoing inputs from the surrounding visual environment. Depending on the situation, one might prioritize mnemonic contents (i.e., remembering details of a past event), or preferentially attend sensory inputs (i.e., minding traffic while crossing a street). Previous fMRI work has shown that early sensory regions can simultaneously represent both mnemonic and passively viewed sensory information. Here we test the limits of such simultaneity by manipulating attention towards sensory distractors during a working memory task performed by human subjects during fMRI scanning. Participants remembered the orientation of a target grating while a distractor grating was shown during the middle portion of the memory delay. Critically, there were several subtle changes in the contrast and the orientation of the distractor, and participants were cued to either ignore the distractor, detect a change in contrast, or detect a change in orientation. Despite sensory stimulation being matched in all three conditions, the fidelity of memory representations in early visual cortex was highest when the distractor was ignored, intermediate when participants attended distractor contrast, and lowest when participants attended the orientation of the distractor during the delay. In contrast, the fidelity of distractor representations was lowest when ignoring the distractor, intermediate when attending distractor-contrast, and highest when attending distractor-orientation. These data suggest a trade-off in early sensory representations when engaging top-down feedback to attend both seen and remembered features and may partially explain memory failures that occur when subjects are distracted by external events.

Keywords: decoding; distraction; fMRI; visual attention; visual working memory.

Figure 1.

(a) Experimental task design. Participants remember the orientation of a brief (0.5s) memory target over a 15s delay, after which they have 3s to rotate a black dial to match the orientation in memory as precisely as possible. On every trial, a distractor grating is shown for 11s during the central-most portion of the delay. And on every trial, this distractor is phase-reversing, and has several small changes to its contrast and orientation. Right before the memory target, participants see a 1.6s cue indicating with 100% validity the upcoming attention condition. Specifically, they need to either (1) ignore the distractor grating, (2) attend and report contrast changes, or (3) attend and report orientation changes. When attending the distractor, participants also report the direction of each change (i.e., whether there is an increase / decrease in contrast, or a clockwise / counterclockwise change in orientation). There is an equal number of trials in each condition, and conditions are randomly interleaved. (b) Behavioral data recorded while participants were in the scanner shows that performance on the distractor attention task is well-matched (t₍₇₎ = 0.157; p = 0.858), and participants perform similarly when detecting contrast (71.85% correct) or orientation (71.56% correct) changes (left panel). Recall of the memory target orientation did differ between conditions (F_(2,14) = 5.889, p = 0.002), and was generally worse when participants had to perform a concurrent orientation attention task on the distractor. Bars indicate average performance, and grey lines individual participants. Asterisks indicate significant post-hoc differences between conditions. (c) Deconvolved BOLD responses for a few example ROI. Distractors in all 3 attention conditions effectively drove univariate responses in early visual areas, with qualitatively higher responses when attention was deployed towards the distractor (shown in purple and pink for attention to distractor contrast or orientation, respectively). In IPS, responses seem more transient, with the strongest response occurring with attention to distractor orientation. Grey background-panels in each subplot indicate the time during which the memory target (0–0.5s, far left panel), the distractor (2.5–13.5s, middle panel), and the response-dial (15.5–18.5s, right most panel) were on the screen. Darker grey lines just above and parallel to the x-axis indicate clusters of consecutive TR’s during which the three attention conditions differ significantly from one another (as calculated with a cluster based permutation test, see Methods).

Figure 2.

(a) Decoding of the target orientation that is held in memory (left) and of the distractor that was physically presented on the screen (right) during the delay period of the main working memory task. The remembered orientation is better decodable when the distractor is ignored (in teal), compared to when it is attended (in purple and pink). The perceived distractor orientation is least decodable when it is ignored (in teal), better decodable when its contrast is attended (in purple), and best decodable when its orientation is attended (in pink). Bars indicate mean orientation decoding averaged across all participants, while light grey lines indicate individual participants. Dark grey lines and asterisks indicate significant post-hoc differences (*p < 0.05; ** p< 0.01; ***p< 0.001) between attention conditions within each ROI (non-parametric t-tests). Note that for decoding of the remembered target, some significance lines are missing an asterisk indicating significance from within-ROI post-hoc t-tests. Due to the lack of an interaction between condition and ROI, these post-hoc t-tests are not technically warranted, and may be prone to type II errors. Rather, these significance lines indicate the post-hoc tests that compare conditions across all ROI, which follow the main effect of attention condition. (b) Same decoding as in (a) for a few example ROI’s, but shown TR-by-TR throughout the trial of the main working memory task. For decoding time courses of all ROI’s, see Supp. Fig. 3. Grey background-panels in each subplot indicate the memory target (0–0.5s, far left panel), distractor (2.5–13.5s, middle panel), and the response (15.5–18.5s, right most panel) periods. Dark grey lines just above and parallel to the x-axis indicate clusters of consecutive TR’s during which the three attention conditions differ significantly from one another (as calculated with a cluster based permutation test, see Methods).

Figure 3.

(a) Decoding of the target orientation that is held in memory, when training a decoder on an independent memory localizer task. The remembered orientation is better represented when the concurrent visual distractor is ignored (teal) than when its contrast (purple) or orientation (pink) are attended. Only in parietal areas (IPS0 and IPS1–3) is there cross-generalization from memory without visual input (i.e., the memory localizer task) to memory with concurrent visual input that is also attended (i.e., the main memory task when the distractor is also attended). (b) Decoding of the sensory distractor in the main memory task, when training a decoder on an independent sensory localizer. Response patterns from the sensory localizer task (which used a blob detection task) generalize to responses to the sensory distractor in the main memory task in most ROI, but not in IPS areas. Possibly because of this, no significant differences between the 3 attention conditions are uncovered. Bars indicate mean orientation decoding averaged across all participants, while light grey lines indicate individual participants. Asterisks in matching condition colors indicate significant post-hoc decoding performance (*p < 0.05; ** p< 0.01; ***p< 0.001) compared to chance.