Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2008 Jan 1;39(1):423-35.
doi: 10.1016/j.neuroimage.2007.08.035. Epub 2007 Aug 31.

Spatiotemporal dynamics of audiovisual speech processing

Lynne E Bernstein  1 , Edward T Auer Jr, Michael Wagner, Curtis W Ponton
Affiliations

Spatiotemporal dynamics of audiovisual speech processing

Lynne E Bernstein et al. Neuroimage. .

Abstract

The cortical processing of auditory-alone, visual-alone, and audiovisual speech information is temporally and spatially distributed, and functional magnetic resonance imaging (fMRI) cannot adequately resolve its temporal dynamics. In order to investigate a hypothesized spatiotemporal organization for audiovisual speech processing circuits, event-related potentials (ERPs) were recorded using electroencephalography (EEG). Stimuli were congruent audiovisual/ba/, incongruent auditory/ba/synchronized with visual/ga/, auditory-only/ba/, and visual-only/ba/and/ga/. Current density reconstructions (CDRs) of the ERP data were computed across the latency interval of 50-250 ms. The CDRs demonstrated complex spatiotemporal activation patterns that differed across stimulus conditions. The hypothesized circuit that was investigated here comprised initial integration of audiovisual speech by the middle superior temporal sulcus (STS), followed by recruitment of the intraparietal sulcus (IPS), followed by activation of Broca's area [Miller, L.M., d'Esposito, M., 2005. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. Journal of Neuroscience 25, 5884-5893]. The importance of spatiotemporally sensitive measures in evaluating processing pathways was demonstrated. Results showed, strikingly, early (<100 ms) and simultaneous activations in areas of the supramarginal and angular gyrus (SMG/AG), the IPS, the inferior frontal gyrus, and the dorsolateral prefrontal cortex. Also, emergent left hemisphere SMG/AG activation, not predicted based on the unisensory stimulus conditions was observed at approximately 160 to 220 ms. The STS was neither the earliest nor most prominent activation site, although it is frequently considered the sine qua non of audiovisual speech integration. As discussed here, the relatively late activity of the SMG/AG solely under audiovisual conditions is a possible candidate audiovisual speech integration response.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Interlip distance and jaw drop in pixels. The vertical 0-msec line corresponds to the onset of the acoustic signal and the 0-msec EEG sampling point.
Figure 2
Figure 2
Mean global field power (MGFP) for the grand mean ERPs in the AO, VO, and the two AV conditions of congruent AV stimulation (auditory /ba/ and visual /ba/) and incongruent AV stimulation (auditory /ba/ and visual /ga/). Each stimulus condition produced two distinct peaks in the MGFP.
Figure 3(a-d)
Figure 3(a-d)
Current density reconstructions for each condition. (a) Left hemisphere from 50 msec to 100 msec in 5-msec steps. Columns are auditory-only (AO) /bα/, audiovisual congruent (AVc) /bα/, audiovisual incongruent (AVi) with auditory /bα/ and visual /gα/, visual-only (VO) /gα/ and VO /bα/. (b) Left hemisphere from 100 to 240 msec in 20-msec steps, with 250 msec, showing that activity was resolved. (c) The right hemisphere from 50 to 100 msec in 5-msec steps. (d) The right hemisphere from 100 to 240 msec in 20-msec steps, with 250 msec, showing that activity was resolved except in the AVi condition.
Figure 3(a-d)
Figure 3(a-d)
Current density reconstructions for each condition. (a) Left hemisphere from 50 msec to 100 msec in 5-msec steps. Columns are auditory-only (AO) /bα/, audiovisual congruent (AVc) /bα/, audiovisual incongruent (AVi) with auditory /bα/ and visual /gα/, visual-only (VO) /gα/ and VO /bα/. (b) Left hemisphere from 100 to 240 msec in 20-msec steps, with 250 msec, showing that activity was resolved. (c) The right hemisphere from 50 to 100 msec in 5-msec steps. (d) The right hemisphere from 100 to 240 msec in 20-msec steps, with 250 msec, showing that activity was resolved except in the AVi condition.
Figure 3(a-d)
Figure 3(a-d)
Current density reconstructions for each condition. (a) Left hemisphere from 50 msec to 100 msec in 5-msec steps. Columns are auditory-only (AO) /bα/, audiovisual congruent (AVc) /bα/, audiovisual incongruent (AVi) with auditory /bα/ and visual /gα/, visual-only (VO) /gα/ and VO /bα/. (b) Left hemisphere from 100 to 240 msec in 20-msec steps, with 250 msec, showing that activity was resolved. (c) The right hemisphere from 50 to 100 msec in 5-msec steps. (d) The right hemisphere from 100 to 240 msec in 20-msec steps, with 250 msec, showing that activity was resolved except in the AVi condition.
Figure 3(a-d)
Figure 3(a-d)
Current density reconstructions for each condition. (a) Left hemisphere from 50 msec to 100 msec in 5-msec steps. Columns are auditory-only (AO) /bα/, audiovisual congruent (AVc) /bα/, audiovisual incongruent (AVi) with auditory /bα/ and visual /gα/, visual-only (VO) /gα/ and VO /bα/. (b) Left hemisphere from 100 to 240 msec in 20-msec steps, with 250 msec, showing that activity was resolved. (c) The right hemisphere from 50 to 100 msec in 5-msec steps. (d) The right hemisphere from 100 to 240 msec in 20-msec steps, with 250 msec, showing that activity was resolved except in the AVi condition.
Figure 4(a-b)
Figure 4(a-b)
Left (4a) and right (4b) hemisphere activation patterns. AVc activity in red, and AVi activity in blue. Time intervals correspond to the resolution in Figure 3(a-d). (Abbreviations: dorsolateral prefrontal cortex, DLPFC; occipital cortex, OC; supramarginal gyrus/angular gyrus, SMG/AG; intraparietal sulcus, IPS; superior temporal gyrus, STG; inferior temporal gyrus, ITG, superior temporal sulcus, STS; inferior frontal, IF).
Figure 4(a-b)
Figure 4(a-b)
Left (4a) and right (4b) hemisphere activation patterns. AVc activity in red, and AVi activity in blue. Time intervals correspond to the resolution in Figure 3(a-d). (Abbreviations: dorsolateral prefrontal cortex, DLPFC; occipital cortex, OC; supramarginal gyrus/angular gyrus, SMG/AG; intraparietal sulcus, IPS; superior temporal gyrus, STG; inferior temporal gyrus, ITG, superior temporal sulcus, STS; inferior frontal, IF).
See this image and copyright information in PMC

References

    1. Amunts K, Schleicher A, Burgel U, Mohlberg H, Uylings HB, Zilles K. Broca's region revisited: cytoarchitecture and intersubject variability. Journal of Comparative Neurology. 1999;412:319–341. - PubMed
    1. Arnold P, Hill F. Bisensory augmentation: A speechreading advantage when speech is clearly audible and intact. British Journal of Psychology. 2001;92:339–355. - PubMed
    1. Beauchamp MS. Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics. 2005;3:93–113. - PMC - PubMed
    1. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A. Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nature Neuroscience. 2004;7:1190–1192. - PubMed
    1. Bernstein LE, Auer ET, Jr., Moore JK. Audiovisual Speech Binding: Convergence or Association? In: Calvert GA, Spence C, Stein BE, editors. Handbook of Multisensory Processing. MIT; Cambridge, MA: 2004a. pp. 203–223.

Publication types