Visual activation and audiovisual interactions in the auditory cortex during speech perception: intracranial recordings in humans

Abstract

Hemodynamic studies have shown that the auditory cortex can be activated by visual lip movements and is a site of interactions between auditory and visual speech processing. However, they provide no information about the chronology and mechanisms of these cross-modal processes. We recorded intracranial event-related potentials to auditory, visual, and bimodal speech syllables from depth electrodes implanted in the temporal lobe of 10 epileptic patients (altogether 932 contacts). We found that lip movements activate secondary auditory areas, very shortly (approximately equal to 10 ms) after the activation of the visual motion area MT/V5. After this putatively feedforward visual activation of the auditory cortex, audiovisual interactions took place in the secondary auditory cortex, from 30 ms after sound onset and before any activity in the polymodal areas. Audiovisual interactions in the auditory cortex, as estimated in a linear model, consisted both of a total suppression of the visual response to lipreading and a decrease of the auditory responses to the speech sound in the bimodal condition compared with unimodal conditions. These findings demonstrate that audiovisual speech integration does not respect the classical hierarchy from sensory-specific to associative cortical areas, but rather engages multiple cross-modal mechanisms at the first stages of nonprimary auditory cortex activation.

Figure 1.

Location of multicontact electrodes in the 10 patients, reported on a common 3D representation of the right hemisphere. The 3D representation of the cortex has been segmented from the anatomical MRI of the right hemisphere of the standard MNI brain. Open and filled circles are electrodes implanted in the left and right hemispheres, respectively. X and Y coordinates of each electrode of each patient have been normalized to the MNI space using the Talairach method. Letters correspond to the name prefix of electrode contacts. The names of left-implanted electrodes are followed by a prime.

Figure 2.

Auditory, visual, and interaction responses recorded from a multicontact electrode (H) passing through the transverse gyrus and the planum temporale in three patients (bipolar data). Each row corresponds to one particular patient. A, Location of the multielectrode relative to a 3D rendering of the cortical surface (superior part of the temporal cortex), segmented from each patient's anatomical MRI. B, Position of the multicontact electrode in the coronal plane containing the electrode. The dark blue lines delineate the cortical surface and the light blue lines the interface between the white and gray matters. C, Spatiotemporal profile of the auditory response along the multicontact electrode axis from 300 ms before to 600 ms after sound onset. D, Spatiotemporal profile of the visual response in the same latency range. E, Spatiotemporal profile of the interaction response (computed as the difference between the bimodal response and the sum of the unimodal responses). The yellow boxes delimit the responses with statistically significant amplitudes [p < 0.00001 for auditory (C) and visual (D) panels; p < 0.001 corrected for multiple comparisons on the temporal dimension for interaction (E) panel]. Yellow arrows indicate the sites and times for which the interaction pattern corresponds to the modulation of the transient auditory response.

Figure 3.

Statistically significant auditory response to the auditory syllables in all patients reported on a common 3D representation of the auditory cortex at 35, 65, 120 and 200 ms after the onset of the auditory stimulus. Each sphere represents the amplitude recorded at one contact in one patient. The radius of the sphere is proportional to the response amplitude, and its color indicates the polarity. Left activations were reported on the right hemisphere. A, Monopolar montage (the voltage values are measured between each contact and a common reference contact) provides information about the polarity of the recorded component. B, Bipolar montage (voltage values are measured between two adjacent contacts); amplitudes in this case indicate that the corresponding neural sources are very close to the site of recording.

Figure 4.

Comparison between (A) intracortically recorded responses to lip movements and (B) fMRI results from lipreading experiments, displayed on a lateral view of the MNI brain temporal cortex. In both cases, spheres stand for activations regardless of the size of the effect found. Left activations were reported on the right hemisphere. A, Statistically significant activations evoked by lip movements alone in our experiment. Each color stands for a cluster of similar responses observed in at least two patients. Numbers in parentheses refer to the type of the visual response as characterized in supplemental Table 2, available at www.jneurosci.org as supplemental material, and described in the text. Individual coordinates have been converted into the MNI space using Talairach method. B, Each color corresponds to a particular contrast that has been tested in studies by the following: (1) Calvert et al. (1997); (2) Puce et al. (1998); (3) Puce and Allison (1999); (4) MacSweeney et al. (2000); (5) Campbell et al. (2001); (6) MacSweeney et al. (2001); (7) Olson et al. (2002); (8) MacSweeney et al. (2002); (9) Paulesu et al. (2003); (10) Calvert and Campbell (2003). When originally given in Talairach space, fMRI activations were converted into the MNI space using the Talairach method.

Figure 5.

Comparison between the location of the visual responses recorded in the auditory cortex and the earliest auditory responses (<30 ms) recorded in the primary auditory cortex. The individual coordinates have been converted to MNI coordinates using the Talairach method and reported on the standard MNI brain. Left activations were reported on the right hemisphere.

Figure 6.

Locations of the interaction responses in the superior temporal cortex. The individual coordinates have been converted to MNI coordinates using the Talairach method and reported on the standard MNI brain. Left activations were reported on the right hemisphere. (*) Indicates that the effect was significant only in the monopolar data for the second type of interaction (see Results, Audiovisual interactions).