Decoding reveals the contents of visual working memory in early visual areas

Abstract

Visual working memory provides an essential link between perception and higher cognitive functions, allowing for the active maintenance of information about stimuli no longer in view. Research suggests that sustained activity in higher-order prefrontal, parietal, inferotemporal and lateral occipital areas supports visual maintenance, and may account for the limited capacity of working memory to hold up to 3-4 items. Because higher-order areas lack the visual selectivity of early sensory areas, it has remained unclear how observers can remember specific visual features, such as the precise orientation of a grating, with minimal decay in performance over delays of many seconds. One proposal is that sensory areas serve to maintain fine-tuned feature information, but early visual areas show little to no sustained activity over prolonged delays. Here we show that orientations held in working memory can be decoded from activity patterns in the human visual cortex, even when overall levels of activity are low. Using functional magnetic resonance imaging and pattern classification methods, we found that activity patterns in visual areas V1-V4 could predict which of two oriented gratings was held in memory with mean accuracy levels upwards of 80%, even in participants whose activity fell to baseline levels after a prolonged delay. These orientation-selective activity patterns were sustained throughout the delay period, evident in individual visual areas, and similar to the responses evoked by unattended, task-irrelevant gratings. Our results demonstrate that early visual areas can retain specific information about visual features held in working memory, over periods of many seconds when no physical stimulus is present.

Figure 1. Design of working memory experiment and resulting time course of fMRI activity

a, Timing of events for an example working memory trial. Two near-orthogonal gratings (25°±3°, 115±3°) were briefly presented in randomized order, followed by a numerical cue (red “1” or green “2”) indicating which grating to remember. After an 11s retention period, a test grating was presented, and subjects reported whether it was rotated clockwise or counterclockwise relative to the cued grating. b, Time course of mean BOLD activity (N=6) in corresponding regions of areas V1–V4 during the working memory task (0–16s) and subsequent fixation period (16–32s). Error bars indicate ±1 S.E.M. Time points 6–10s (shaded grey area) were averaged for subsequent decoding analysis of delay-period activity. The start of this time window was chosen to allow for peak BOLD activity to fully emerge; we selected a conservative end point of 10s to exclude any potential activity elicited by the test grating.

Figure 2. Orientation decoding results for areas V1-V4

Accuracy of orientation decoding for remembered gratings in the working memory experiment (green circles), unattended presentations of low-contrast gratings (red triangles), and generalization performance across the two experiments (black squares). Error bars indicate ±1 S.E.M. Decoding was applied to the 120 most visually responsive voxels in each of V1, V2, V3, and V3A/V4 (480 voxels for V1-V4 pooled), as determined by their responses to a localizer stimulus (1-4°eccentricity). Areas V3A and V4 showed similar decoding performance but had fewer available voxels, so these regions were combined.

Figure 3. Time-resolved decoding of individual fMRI time points

Orientation decoding of unattended stimulus gratings (red triangles) and remembered gratings during working memory (green circles), for activity obtained from areas V1-V4 (a) and V1 only (b). Note that orientation information persists throughout the delay period during the working memory task, up until presentation of the test grating at time of 13s. Error bars indicate ±1 S.E.M.