The Interplay Between Multisensory Processing and Attention in Working Memory: Behavioral and Neural Indices of Audiovisual Object Storage

Abstract

Although real-life events are multisensory, how audio-visual objects are stored in working memory is an open question. At a perceptual level, evidence shows that both top-down and bottom-up attentional processes can play a role in multisensory interactions. To understand how attention and multisensory processes interact in working memory, we designed an audiovisual delayed match-to-sample task in which participants were presented with one or two audiovisual memory items, followed by an audiovisual probe. In different blocks, participants were instructed to either (a) attend to the auditory features, (b) attend to the visual features, or (c) attend to both auditory and visual features. Participants were instructed to indicate whether the task-relevant features of the probe matched one of the task-relevant feature(s) or objects in working memory. Behavioral results showed interference from task-irrelevant features, suggesting bottom-up integration of audiovisual features and their automatic encoding into working memory, irrespective of task relevance. Yet, event-related potential analyses revealed no evidence for active maintenance of these task-irrelevant features, while they clearly taxed greater attentional resources during recall. Notably, alpha oscillatory activity revealed that linking information between auditory and visual modalities required more attentional demands at retrieval. Overall, these results offer critical insights into how and at which processing stage multisensory interactions occur in working memory.

Keywords: EEG; alpha oscillations; multisensory processing; selective attention; working memory.

FIGURE 1

Schematic illustration of one versus two‐item trials. (A) In one‐item trials, the first item was an audiovisual filler item; (B) while in two‐item trials, this was replaced by a memory item. The three memory conditions only differed in terms of their task instructions, while the physical stimulation was always the same. ISI, inter‐stimulus interval; ITI, inter‐trial interval.

FIGURE 2

Illustration of probe types and respective proportion of trials per memory condition. (A) In single‐feature conditions, congruent and incongruent probe types appeared equally often; (B) to equalize “yes” and “no” responses across probe types in the conjunction condition, the “auditory match + visual match” probe type was presented in half of the conjunction condition trials, while the other three probe conditions were equally distributed across the remaining 50% of trials.

FIGURE 3

Probe congruency effect between single‐feature conditions for different set size trials. (A) Proportion of correct responses for the congruency effect (congruent—incongruent) between auditory and visual conditions; (B) RTs for the congruency effect between auditory and visual conditions. Boxplots show the ±1.5 interquartile range and the median. The dots illustrate individual participant averages per condition. A black cross illustrates the condition mean. *p < 0.05.

FIGURE 4

Set size effect across memory conditions. (A) The upper row shows how many task‐relevant features each set size (i.e., set size 1 vs set size 2) included per condition. The lower row illustrates how many objects each set size included per condition; (B) The proportion of correct responses varied by set size across memory conditions; (C) Reaction times varied by set size across memory conditions. Boxplots show the ±1.5 interquartile range and the median. The dots illustrate individual scores per condition. A black cross illustrates the condition mean. *p < 0.05, ***p < 0.001, n.s., not significant.

FIGURE 5

Set size effect between single‐feature and conjunction conditions. (A) Proportion of correct responses for the set size effect (load 1–load 2) between single feature and conjunction conditions; (B) RTs for the set size effect between single feature and conjunction conditions. Boxplots show the ±1.5 interquartile range and the median. The dots illustrate individual participant averages per condition. A black cross illustrates the condition mean. *p < 0.05, n.s., not significant.

FIGURE 6

Condition‐specific, grand‐average ERP waveforms at anterior (SAN) and posterior (NSW) electrode sites. Mean ERP amplitudes in (A) and (B) show the SAN and NSW amplitudes during the maintenance interval for each memory condition for set sizes 1 and 2 trials (error bars show standard deviations). The topographies show the electrode clusters used for SAN (left panel) and NSW (right panel). The ERPs in (C) and (D) depict the time course of the SAN and NSW, respectively, separately for the auditory, visual, and conjunction conditions and for set sizes 1 and 2 trials. The gray rectangle indicates the corresponding analysis time window, spanning the maintenance period. Significant paired‐sample t‐test contrasts (i.e., auditory set size 1 vs. set size 2 and conjunction set size 1 versus set size 2 comparisons for SAN and auditory set size 1 versus set size 2 for NSW) were marked with an asterisk (p < 0.05). n.s., not significant.

FIGURE 7

Cluster‐based permutation test results between memory conditions. Differences in oscillatory power between conditions were tested with cluster‐based permutation tests across (A) the midcentral electrode cluster and (B) the parieto‐occipital electrode cluster. Solid lines illustrate significant clusters with a t‐mass value smaller than 1st or larger than the 99th percentile of the distribution of significance probabilities of the null distribution.

FIGURE 8

Cluster‐based permutation test results for set size 2 versus 1 contrast across memory conditions. Pair‐wise contrasts (between set size 2 vs. set size 1 trials) for time‐frequency power between conditions were tested with cluster‐based permutation tests for (A) the mid‐central electrodes and for (B) the parieto‐occipital electrodes. Solid lines illustrate significant clusters with a t‐mass value smaller than 1st or larger than the 99th percentile of the distribution of significance probabilities of the null distribution.

FIGURE 9

Time‐frequency modulations and cluster‐based permutation test results for incongruent versus congruent probe contrast. (A) Power was averaged across the midcentral electrode cluster for congruent and incongruent trials (upper and middle rows). Differences between the two conditions were tested with cluster‐based permutation tests (lower row); (B) Power was averaged across the parieto‐occipital electrodes for congruent and incongruent trials (upper and middle rows). Differences between the two conditions were tested with cluster‐based permutation tests (lower row). Solid lines illustrate significant clusters with a t‐mass value smaller than 1st or larger than the 99th percentile of the distribution of significance probabilities of the null distribution.