The neural basis of swap errors in working memory

Abstract

When making decisions in a cluttered world, humans and other animals often have to hold multiple items in memory at once - such as the different items on a shopping list. Psychophysical experiments in humans and other animals have shown remembered stimuli can sometimes become confused, with participants reporting chimeric stimuli composed of features from different stimuli. In particular, subjects will often make "swap errors" where they misattribute a feature from one object as belonging to another object. While swap errors have been described behaviorally, their neural mechanisms are unknown. Here, we elucidate these neural mechanisms through trial-by-trial analysis of neural population recordings from posterior and frontal brain regions while monkeys perform two multi-stimulus working memory tasks. In these tasks, monkeys were cued to report the color of an item that either was previously shown at a corresponding location (requiring selection from working memory) or will be shown at the corresponding location (requiring attention to a position). Animals made swap errors in both tasks. In the neural data, we find evidence that the neural correlates of swap errors emerged when correctly remembered information is selected incorrectly from working memory. This led to a representation of the distractor color as if it were the target color, underlying the eventual swap error. We did not find consistent evidence that swap errors arose from misinterpretation of the cue or errors during encoding or storage in working memory. These results suggest an alternative to established views on the neural origins of swap errors, and highlight selection from and manipulation in working memory as crucial - yet surprisingly brittle - neural processes.

Figure 1:

The retrospectively and prospectively cued working memory tasks, as well as the behavioral modeling approach. a Schematic of the retrospective task. b Schematic of the prospective task, note that the order of the cue and stimuli is switched with respect to the respective task shown in a. c Schematic of the possible trial outcomes: correct response (close to the target), a guess (uniform random), and a swap (close to the distractor). d Schematic of the behavioral modeling approach. Each trial response is explained in terms of three parameters: the rate of swap errors, the rate of guesses, and the dispersion around the target response. A single model is fit hierarchically across all the sessions, where the session level parameters (middle) are modeled as being sampled from a distribution of animal-level parameters (top). The model is fit separately for each animal. e Histograms of the errors from the target (left) and the distractor stimulus (right) for Monkey E (top) and Monkey W (bottom) on retrospective trials. The dashed black line is the posterior predictive distribution from the behavioral model. The concentration around zero for the plots on the right indicate that both monkeys make swap errors. f The inferred rate of each response type on retrospective trials for each session (points) and both monkeys (colors). g An example session from Monkey E on the retrospective task. Each point is the response on a particular trial. The triangle is the three-dimensional simplex for the probabilities of each response type. Each of the three corners represent probability 1 of a particular response type. Colored points have probability > .5 of belonging to their respective response type. h - j The same as e - g but for prospective trials, and the example session in j is from Monkey W.

Figure 2:

The neural correlates of swap errors in the retrospectively cued working memory task. a Schematic of an example trial during the first delay period of the retrospective task. The nominal (gray star) and misbound (purple star) representations are shown for this example trial. The misbound representation manifests as a swapped representation of the upper and lower colors (green-upper, blue-lower to blue-upper, green-lower). b The average (top bar) and aggregate posterior (distribution) across sessions for the ${\mathrm{p}}_{\text{misbind}}$ parameter in the neural mixture model for the first delay period. Values close to one indicate an association between swap errors and misbound representations, values close to zero indicate that trials with swap errors are still associated with the nominal representation. c Schematic of the cross-validated version of the analysis. Nominal and misbound representations (stars) are constructed for each trial using the same kind of linear model as used in the neural mixture model. The model is fit only on likely correct trials. Then, held-out correct trials and swap trials (circles) are projected onto the dimension connecting the nominal and misbound representations (dashed line). The distance is normalized to be between zero and one. d Time course of evidence for an association between swap errors and the misbound representation under the analysis in c, for each monkey (top and bottom) as well as centered on the time of sample (left) and cue (right) presentation. e Schematic of the same example trial from above, showing the contrast between the nominal representation (gray stars) and the mis-selected colors representation (top, pink star) as well as the mis-interpreted cue representation (bottom, orange star). f The average (contour outline) and aggregate posterior (heatmap) evidence for the nominal, mis-selected colors, and mis-interpreted cue representations under the neural mixture model in delay 2. g Schematic of the cross-validated version of the analysis. Correct, color selection, and cue interpretation prototypes are constructed using a linear model fit on likely correct trials. Then, held-out correct trials and likely swap trials are projected along the axes connecting the correct prototype with each of the error prototypes and normalized as before. h Time course of evidence for the mis-selected colors and mis-interpreted cue representations for both monkeys (top and bottom) as well as centered on the cue presentation (left) and appearance of the response wheel (right). i The difference between the evidence for an association between swap errors and the alternative representations in delay 1 and delay 2 under the neural mixture model for each session (left) and averaged across all sessions (right; positive values indicate more evidence for correlates in delay 2).

Figure 3:

The neural correlates of swap errors in the prospectively cued working memory task. a Schematic of an example trial in the prospective task, showing the nominal (gray star) and mis-interpreted cue (orange star) representations. b Schematic of the decoder generalization analysis used in c and f. A linear decoder is trained to discriminate between trials with an upper or lower cue using only likely correct trials (left). Then, that same decoder is tested on likely swap trials (right). c The performance of a decoder trained and tested during delay 1 on likely correct trials (left x-axis) and trained on likely correct but tested on likely swap trials (left y-axis), for each session (points) and the difference between the two averaged across sessions (right). d Schematic of an example trial in the prospective task, showing the nominal as well as the mis-bound color (top, purple star) and mis-selected cue (bottom, pink star) representations. e The average (contour outline) and aggregate posterior evidence for the association of the nominal, misbound, and mis-selected representations with swap errors under the neural mixture model. f The analysis schematized in b applied to the second delay period. g The same decoding approach schematized in b and applied to delay 1 and 2 in c and f, respectively. Here, we compare across delay periods for likely correct trials (left) and likely swap trials (right).