Updating the dual-mechanism model for cross-sensory attentional spreading: The influence of space-based visual selective attention

Abstract

Selective attention to visual stimuli can spread cross-modally to task-irrelevant auditory stimuli through either the stimulus-driven binding mechanism or the representation-driven priming mechanism. The stimulus-driven attentional spreading occurs whenever a task-irrelevant sound is delivered simultaneously with a spatially attended visual stimulus, whereas the representation-driven attentional spreading occurs only when the object representation of the sound is congruent with that of the to-be-attended visual object. The current study recorded event-related potentials in a space-selective visual object-recognition task to examine the exact roles of space-based visual selective attention in both the stimulus-driven and representation-driven cross-modal attentional spreading, which remain controversial in the literature. Our results yielded that the representation-driven auditory Nd component (200-400 ms after sound onset) did not differ according to whether the peripheral visual representations of audiovisual target objects were spatially attended or not, but was decreased when the auditory representations of target objects were presented alone. In contrast, the stimulus-driven auditory Nd component (200-300 ms) was decreased but still prominent when the peripheral visual constituents of audiovisual nontarget objects were spatially unattended. These findings demonstrate not only that the representation-driven attentional spreading is independent of space-based visual selective attention and benefits in an all-or-nothing manner from object-based visual selection for actually presented visual representations of target objects, but also that although the stimulus-driven attentional spreading is modulated by space-based visual selective attention, attending to visual modality per se is more likely to be the endogenous determinant of the stimulus-driven attentional spreading.

Keywords: cross-modal spread of attention; modality-based; object-based; representation-driven; space-based; stimulus-driven; visual selective attention.

FIGURE 1

(a) Task paradigm shown for trials in a block when line drawings of dogs on the left side were the targets. Each trial consisted of a 200‐ms stimulus presentation and an inter‐trial interval (ITI) of 1,000–1,200 ms. Visual stimulus could be one of the nine unique drawings (three dogs, three cars, and three drums), which was presented randomly to the left or right visual field. Auditory stimulus could be one of the nine unique natural sounds (three barks of dogs, three beeps of cars, and three beats of drums), which always came from the center location. The unilateral drawings (V) and central sounds (A) could be either presented alone or presented synchronously (AV) to form semantically congruent pairs (e.g., barks with dogs), resulting in three main stimulus types. The task for subjects was to make a button‐press in response to the second of two consecutively presented drawings of the target object category (dogs or cars) appearing at the to‐be‐attended spatial location (left or right side), while ignoring all drawings at the unattended side and all sounds if delivered. (b) Schematic diagram of the experimental comparisons used for isolating the cross‐modal attentional spreading that originated from the representation‐driven process (left) and the stimulus‐driven process (right). The superscripts “+” and “−” denote visual or auditory representations of the target and nontarget objects, respectively, which applies to all relevant figures and paragraphs in the main text

FIGURE 2

(a) ERP waveforms elicited by peripheral visual stimuli as functions of visual‐spatial attention (attended, unattended) and target condition (target object [V⁺], nontarget object [V⁻]), which were averaged over the contralateral (relative to the side of visual stimuli) part of the posterior ROI (i.e., P6, P8, PO4, and PO8). The shaded areas on waveforms depict the time windows within which the P1 (90–120 ms), N1 (160–190 ms), and SN (240–290 ms) components were quantified, respectively. (b) Scalp topographies, with contralateral voltages being projected to the right hemisphere and ipsilateral to the left, are shown for the spatially attended minus unattended mean difference amplitudes within the P1 and N1 intervals (first and second columns), and for the target minus nontarget mean difference amplitudes within the SN interval (third column). The white dots on scalp topographies depict the whole posterior ROI (P5, P7, PO3, PO7; P6, P8, PO4, and PO8) over which each component was measured. Space‐based visual attention effects (P1 and N1 modulations) were independent of object‐based visual attention, but object‐based visual attention effect (SN amplitude) was strongly modulated by visual‐spatial attention

FIGURE 3

The extracted auditory ERP waveforms to audiovisual stimuli when their visual constituents were the target (red traces: A⁺V⁺ − V⁺) and nontarget (blue traces: A⁻V⁻ − V⁻) objects, plotted separately for (a) visual‐spatially attended and (b) unattended conditions. (c) ERP waveforms elicited by auditory‐only stimuli when their representations were corresponding to target (red trace: A⁺) and nontarget (blue trace: A⁻) objects. These ERP waveforms were averaged over the fronto‐central ROI (F1, Fz, F2, FC1, FCz, FC2, C1, Cz, and C2). The shaded areas on waveforms depict the four time windows (200–250 ms, 250–300 ms, 300–350 ms, and 350–400 ms) within which the Nd component was quantified. Scalp topographies (depicting voltages contralateral to the side of visual constituents on the right hemisphere and ipsilateral on the left for AV stimuli), are shown as the target minus nontarget mean difference amplitudes within each Nd interval for each condition. The white dots on topographies depict the fronto‐central ROI over which the Nd component was measured. For AV stimuli (a,b), the representation‐driven Nd component, indexed by significantly larger negative amplitude for target than nontarget objects, was prominent and sustained independent of space‐selective visual attention; for A‐only stimuli (c), the Nd amplitude was decreased and less sustained. *: p <.05 for the target versus nontarget contrast

FIGURE 4

Left: The extracted‐auditory ERP waveforms to visual‐spatially attended (blue trace) and unattended (green trace) audiovisual stimuli when their visual constituents were the nontarget objects, and ERP waveforms evoked by auditory‐only stimuli (red traces) when their representations were corresponding to the nontarget objects. These ERP waveforms were averaged over the same fronto‐central ROI as Figure 3, and shaded areas on waveforms depict the same successive Nd intervals as Figure 3. Right: Scalp topographies are shown for the extracted‐auditory minus auditory‐only difference amplitudes during each Nd interval separately for visual‐spatially attended [(A⁻V⁻ _att − V⁻ _att) − A⁻] and unattended [(A⁻V⁻ _unatt − V⁻ _unatt) − A⁻] conditions, with voltages contralateral to the side of visual constituents being projected to the right hemisphere and ipsilateral to the left. Although the stimulus‐driven Nd component, indexed by significantly greater negative amplitude on the extracted auditory than auditory‐only nontarget ERP waveforms, was larger when the visual constituents of audiovisual nontarget objects were spatially attended than unattended, it was still evident to some extent when the visual constituents of audiovisual nontarget objects were spatially unattended. *: p <.05 for the extracted auditory versus auditory‐only contrast