Output planning at the input stage in visual working memory

Abstract

Working memory serves as the buffer between past sensations and future behavior, making it vital to understand not only how we encode and retain sensory information in memory but also how we plan for its upcoming use. We ask when prospective action goals emerge alongside the encoding and retention of visual information in working memory. We show that prospective action plans do not emerge gradually during memory delays but are brought into memory early, in tandem with sensory encoding. This action encoding (i) precedes a second stage of action preparation that adapts to the time of expected memory utilization, (ii) occurs even ahead of an intervening motor task, and (iii) predicts visual memory-guided behavior several seconds later. By bringing prospective action plans into working memory at an early stage, the brain creates a dual (visual-motor) memory code that can make memories more effective and robust for serving ensuing behavior.

Fig. 1. Task, behavioral performance, and hypothetical scenarios (experiment 1).

(A) Task schematic of experiment 1. Trials started with a precue that could be either informative (80%; the color change of the fixation cross indicated with 100% validity which item would be probed) or neutral (20%; the cross turned gray, such that both memory items were equally likely to be probed). After a predictable memory delay of 2 or 4 s (blocked), participants reproduced the tilt of the memory item that was indicated by the color change of the central fixation cross. Item tilt was directly linked to the required action, such that a left/right-tilted item required reproduction with the left/right response hand (while required response duration depended on the detailed memory representation of tilt magnitude). Item location and required response hand were manipulated orthogonally. The example trials depict two left-item trials that required either a right (top) or left (bottom) hand response at the end of the memory delay. CTI, cue-target interval. (B) Reproduction errors and response times as a function of precue informativeness and memory delay. (C) Average response density as a function of the reported tilt and the tilt of the probed item. Zero degrees denote vertical and negative (positive) values denote a leftward (rightward) tilt. Error bars and shadings indicate ±1 SEM calculated across participants. Gray lines in (B) denote individual participants (n = 25). (D and E) Hypothetical patterns of prospective action preparation as a function of time of expected memory utilization.

Fig. 2. Prospective action preparation co-occurs with visual working memory.

(A) Lateralized neural activity relative to the visual location of the cued (left) and probed (right) memory item. Visual lateralization was calculated in canonical visual electrodes (PO7/PO8). (B) Lateralized neural activity relative to the prospective response hand associated with the cued (left) and probed (right) memory item. Motor lateralization was calculated in canonical motor electrodes (C3/C4). Note how the visual location (visual) and prospective response hand (motor) were independently manipulated, yielding independent neural contrasts between (A) and (B). The black outlines indicate significant clusters following a cluster-based permutation analysis (63). (C) Sensor- and source-level lateralization relative to the visual location of the cued memory item (left, visual alpha) and to the prospective manual action associated with this item (middle, mu-alpha; right, mu-beta). Data from the 400- to 800-ms window after encoding onset (fig. S1 for a time-resolved topographical analysis that also includes the postprobe windows). Contralateral (contra) versus ipsilateral (ipsi) contrast values were projected into the right sensor/hemisphere of each symmetrical electrode/voxel pair, to match the corresponding time-frequency contrasts. Brain-slice depth runs from top left to bottom right in 12 steps. For visualization, a masking threshold was applied to all source values smaller than 50% of the maximum absolute contrast value.

Fig. 3. Early encoding followed by gradual preparation of prospective action.

(A) Time courses of lateralized (effector-specific) action preparation signatures in mu-alpha (left) and mu-beta (right) separated according to the time of expected memory utilization and plotted up to the time of probe onset (2 and 4 s, respectively). Lateralization linked to action preparation was calculated in canonical motor electrodes (C3/C4). Horizontal lines indicate significant clusters, with the black lines denoting the difference between the expect-early and expect-late conditions (all clusters, P < 0.05). Shadings indicate ±1 SEM calculated across participants (n = 25). (B) Topographies at representative time windows in (A).

Fig. 4. Action encoding occurs despite an intervening visual-motor task (experiment 2).

(A) Task schematic of experiment 2. Participants performed a dual task with a primary memory task (like in experiment 1) and an intervening visual-motor task in the memory delay. The intervening task occurred on each trial and always consisted of two left and two right sequentially presented visual squares that indicated participants should press the corresponding button on the keyboard (using the same fingers and buttons that were used for the primary memory task). The first visual stimulus of the intervening task always occurred 1500 ms after encoding onset, while the last visual stimulus always occurred 4500 ms after encoding onset [as marked in (C)]. (B) Hypothetical patterns of action preparation in this task. (C) Neural lateralization in motor electrodes C3/C4 relative to the prospective response hand associated with the cued memory item in the primary memory task, across all frequencies (top), as well as for the 8- to 30-Hz band that spanned both the mu-alpha and the mu-beta bands (bottom). Insets show topographies at representative time windows. The black outlines and blue horizontal lines indicate significant clusters (all clusters, P < 0.05).

Fig. 5. Action encoding reflects the anticipated task, not the visual stimulus.

(A) Schematic of the working memory task and the task-free visual localizer. The circled item in the task schematic indicates the cued memory item. (B) Neural lateralization in visual electrodes relative to the location of the visual item in the task (left) and in the task-free localizer (right). Data from experiments 1 (top) and 2 (bottom). (C) Neural lateralization in motor electrodes relative to the tilt of the visual item in the task (left) and in the task-free localizer (right) in experiment 1 (top) and 2 (bottom). In the task, visual tilt was linked to the prospective response hand, whereas this was not the case in the task-free localizer. Visual localizer modules occurred in between task blocks and contained sequentially presented left and right bars with left and right tilts; identical to the items used in the task, except presented in isolation. The black outlines indicate significant clusters (all clusters, P < 0.05). Conventions as per Figs. 2 and 4.

Fig. 6. Action encoding predicts response times of memory-guided behavior after a multisecond delay.

(A) Prospective action preparation (8- to 30-Hz lateralization in C3/C4 relative to the prospective manual action) as a function of response times after the memory delay (median split), separately for experiment 1 for blocks with a 2-s delay (top), a 4-s delay (middle), and experiment 2 (6.5-s delay, bottom). Mean response times for fast and slow trials were as follows: experiment 1 (2-s delay), 430 ± 19 (fast) and 848 ± 59 ms (slow) (means ± SEM); experiment 1 (4-s delay), 440 ± 18 and 813 ± 48 ms; experiment 2, 337 ± 17 and 668 ± 44 ms. (B) Action encoding for fast and slow trials, averaged across the 400- to 800-ms action-encoding window. This window was chosen a priori (based on the preceding results) and is also indicated in the gray shadings in (A). Error bars and shadings indicate ±1 SEM calculated across participants. Gray lines in (B) denote individual participants (n = 25).

Fig. 7. Action encoding predicts the precision of memory-guided behavior when working memory is interrupted by an intervening task.

(A) Prospective action preparation (8- to 30-Hz lateralization in C3/C4 relative to the prospective manual action) as a function of the precision of the reproduction report in the working memory task at the end of the delay (median split) in experiment 2. Mean reproduction errors for precise and imprecise trials were 12.4° ± 1.6° and 29.6° ± 1.8°. (B) Action encoding for precise and imprecise trials, averaged across the 400- to 800-ms action-encoding window. This window was chosen a priori (based on the preceding results) and is also indicated in the gray shadings in (A). Error bars and shadings indicate ±1 SEM calculated across participants. Gray lines in (B) denote individual participants (n = 25).