Transformed Visual Working Memory Representations in Human Occipitotemporal and Posterior Parietal Cortices

Abstract

Recent fMRI studies reported transformed representations between perception and visual working memory (VWM) in the human early visual cortex (EVC). This is inconsistent with the still widely cited original proposal of the sensory account of VWM, which argues for a shared perception-VWM representation based on successful cross-decoding of the two representations. Although cross-decoding was usually lower than within-VWM decoding and consistent with transformed VWM representations, this has been attributed to experimental differences between perceptual and VWM tasks: once they are equated, the same representation is expected to exist in both. Including human participants of both sexes, this study compared target and distractor representations during the same VWM delay period for the same objects, thereby equating experimental differences. Even with strong VWM representations present throughout the occipitotemporal cortex (OTC, including EVC) and posterior parietal cortex (PPC), fMRI cross-decoding revealed significant representational differences between distractors (perception) and targets (VWM) in both regions. Similar differences existed between target encoding (perception) and delay (VWM), being greater in OTC than PPC, indicating more invariant target representations in PPC than OTC. As only part of the sensory input is usually task-relevant, sustaining sensory input in VWM without selection/refinement/consolidation is both taxing and unnecessary. Transformed representations, mediated by task goals and associative areas coding task-relevant information (e.g., PPC), can easily account for these and other recent findings. A task-driven transformed account of VWM thus better captures the nature of VWM representation in the human brain (including EVC) than the sensory representations originally proposed by the sensory account of VWM.

Keywords: fMRI; occipitotemporal cortex; posterior parietal cortex; vision; visual object representation; visual short-term memory; visual working memory.

Figure 1.

Experimental design and example brain regions of interest (ROIs). A, Example trials showing the trial sequence. In each trial, a single target is shown; after an extended delay period filled with either a blank screen or a sequential presentation of exemplars of another object, a probe is shown. The probe was either an exact match or a different exemplar of the same object type. The entire image set is available from the online data deposit. B, The four types of objects used. C, Example ROIs shown on the inflated cortical surfaces. Reproduced from Vaziri-Pashkam and Xu (2019). D, Decoding accuracies during the VWM delay period for targets in trials without and with distractors and for distractors. The colored symbols above the bars mark the decoding significance of each bar compared with chance (0.5). Error bars indicate SEM. *p < 0.05, **0.01 < p < 0.001, ***p < 0.001.

Figure 2.

Cross-distractor decoding of VWM targets for trials with distractors. A, An illustration of within-target and cross-distractor target decoding during the delay period. A classifier was either trained by VWM targets or by distractors to decode VWM targets. The distractors during VWM target training and the targets during distractor training were matched (e.g., couches in the examples shown). See Extended Data Figure 2-1 for additional results showing that the same results were obtained regardless of whether or not the irrelevant object remained the same or different in distractor training and target testing. B, Results of A for all the ROIs (left) and the three ROI sectors (right). C, Same as A, except target training and decoding occurred during the encoding period. Distractor training still occurred during the delay period. D, Results of C for all the ROIs (left) and the three ROI sectors (right). E, Cross-decoding ratios for both types of cross-decoding for all the ROIs. F, The same results as in E for the three ROI sectors. In B and D, the colored symbols above the bars mark the decoding significance of each bar compared with chance (0.5). The black symbols right above the colored symbols mark the significance of decoding drop between within and cross-decoding. In E and F, the colored symbols above the bars mark the significance of the ratio compared with 1. The black symbols right above the colored symbols mark the significance of the ratio difference between the two types of cross-decoding. In B, D, and F, the black symbols above the brackets mark the significance of the difference in either cross-decoding drop (B and D) or ratio for the two types of cross-decoding (F) between pairs of ROI sectors. Error bars indicate SEM. *p < 0.05, **0.01 < p < 0.001, ***p < 0.001.

Figure 3.

Comparing cross-distractor decoding of VWM targets for trials with and without distractors. A, An illustration of within-target and cross-distractor target decoding during the delay period for trials with distractors (same as in Fig. 2A). B, Same as A, but for trials without distractors. C, Results of A and B, plotted for separately each ROI. D, Same as C, for the three ROI sectors. In C and D, the colored symbols above the bars mark the decoding significance of each bar compared with chance (0.5). The black symbols above the colored symbols mark the significance of the decoding drop between within and cross-decoding. Error bars indicate SEM. *p < 0.05, **0.01 < p < 0.001, ***p < 0.001.

Figure 4.

Within and cross-decoding between VWM encoding and delay periods. A, An illustration of within and cross-decoding between encoding and delay periods for trials with distractors. B, Results of A for all the ROIs (left) and the three ROI sectors (right). C, An illustration of within and cross-decoding between encoding and delay periods for trials without distractors. D, Results of C for all the ROIs (left) and the three ROI sectors (right). E, Cross-decoding ratios for both types of trials for all the ROIs. F, The same results for the three ROI sectors. In B and D, the colored symbols above the bars mark the decoding significance of each bar compared with chance (0.5). The black symbols right above the colored symbols mark the significance of decoding drop between within and cross-decoding. In E and F, the colored symbols above the lines/bars mark the significance of the ratio compared with 1. The black symbols right above the colored symbols mark the significance of the ratio difference between the two types of trials. In B, D, and F, the black symbols above the brackets mark the significance of the difference in either cross-decoding drop (B and D) or overall ratio (F) between pairs of ROI sectors. Error bars indicate SEM. *p < 0.05, **0.01 < p < 0.001, ***p < 0.001.