The role of the posterior superior temporal sulcus in audiovisual processing

Abstract

In this study we investigate previous claims that a region in the left posterior superior temporal sulcus (pSTS) is more activated by audiovisual than unimodal processing. First, we compare audiovisual to visual-visual and auditory-auditory conceptual matching using auditory or visual object names that are paired with pictures of objects or their environmental sounds. Second, we compare congruent and incongruent audiovisual trials when presentation is simultaneous or sequential. Third, we compare audiovisual stimuli that are either verbal (auditory and visual words) or nonverbal (pictures of objects and their associated sounds). The results demonstrate that, when task, attention, and stimuli are controlled, pSTS activation for audiovisual conceptual matching is 1) identical to that observed for intramodal conceptual matching, 2) greater for incongruent than congruent trials when auditory and visual stimuli are simultaneously presented, and 3) identical for verbal and nonverbal stimuli. These results are not consistent with previous claims that pSTS activation reflects the active formation of an integrated audiovisual representation. After a discussion of the stimulus and task factors that modulate activation, we conclude that, when stimulus input, task, and attention are controlled, pSTS is part of a distributed set of regions involved in conceptual matching, irrespective of whether the stimuli are audiovisual, auditory-auditory or visual-visual.

Keywords: amodal; audiovisual binding; conceptual integration; congruency; crossmodal.

Figure 1.

Anatomical location of our ROI (in red) relative to other amodal processing areas. Red = areas associated with audiovisual integration in pSTS (Calvert et al. 2000; Olson et al. 2002; Saito et al. 2003; Sekiyama et al. 2003; Wright et al. 2003; Beauchamp, Lee, et al. 2004; Macaluso et al. 2004; van Atteveldt et al. 2004; Miller and D'Esposito 2005; Ojanen et al. 2005; Taylor et al. 2006; Hein et al. 2007; Kreifelts et al. 2007; van Atteveldt, Formisano, Blomert, et al. 2007; van Atteveldt, Formisano, Goebel, et al. 2007). Green = areas associated with audiovisual integration in aSTS (Calvert et al. 1999; Sestieri et al. 2006; van Atteveldt et al. 2004; Noppeney et al. 2007; van Atteveldt, Formisano, Blomert, et al. 2007; van Atteveldt, Formisano, Goebel, et al. 2007). Blue = areas associated with action and tool processing (Phillips et al. 2002; Vandenberghe et al. 2002; Kellenbach et al. 2003; Narain et al. 2003; Emmorey et al. 2004; Lewis et al. 2005; Noppeney et al. 2005, 2007).

Figure 2.

Stimulus trials for Experiments 1a, 1b, and 2. In Experiments 1a and 1b each trial consisted of 2 simultaneously presented audio and visual stimuli, followed immediately by 2 further audio and visual stimuli with a fixation cross presented between trials. In Experiment 1a and 2, a key-press response was made as soon as subjects could determine whether the stimuli referred to the same concept or not. In Experiment 1b subjects alternated a left and right key-press response. All Experiments had 6 trials per block. Note stimuli were presented in their appropriate colors (not shown in this grayscale figure).

Figure 3.

Common activation for crossmodal and intramodal matching. Increased activation for sequential matching relative to fixation before (a) and after (b) removing sensorimotor activation in Experiment 1b (P < 0.5). Across all 3 conditions, the peak co-ordinate in left pSTS was centered at [−50, −50, 10]. In the right, the peak co-ordinate was centered for AV and VV at [54, −54, 8] and for AA at [54, −54, 6]. To highlight pSTS, activation is shown at P < 0.05 corrected for multiple comparisons when the baseline was fixation (a) and P < 0.001 uncorrected on sagittal sections (x = −50) when the baseline removed sensorimotor processing (b). A white circle highlights pSTS activation. Details of all other activation peaks are provided in Supplementary Materials 2.

Figure 4.

Incongruent > congruent matching of simultaneously presented audiovisual objects. To demonstrate the extent of significant activations only, the statistical threshold was set to P < 0.001, uncorrected with a minimum cluster of 35 voxels.

Figure 5.

Effect sizes in left and right pSTS for all conditions in each experiment. Effect sizes (and variance) in the left [−60, −52, 14] and right [64, −48, 12] pSTS show the greater activation for incongruent than congruent trials for each condition. The main effect of incongruency was significant in the simultaneous matching conditions (Experiment 2; Z = 5.0) but not in the sequential matching conditions (Experiment 1; Z < 1.96). See Table 3 for more statistical details.

Figure 6.

Comparison of audiovisual matching in Experiments 1 and 2. Audiovisual matching > fixation for (a) simultaneous and (b) sequential audiovisual presentation. Rendered at P < 0.05, corrected for multiple comparisons. There was no significant difference in pSTS activation when (a) and (b) were directly compared.