Audiovisual integration in macaque face patch neurons

Abstract

Primate social communication depends on the perceptual integration of visual and auditory cues, reflected in the multimodal mixing of sensory signals in certain cortical areas. The macaque cortical face patch network, identified through visual, face-selective responses measured with fMRI, is assumed to contribute to visual social interactions. However, whether face patch neurons are also influenced by acoustic information, such as the auditory component of a natural vocalization, remains unknown. Here, we recorded single-unit activity in the anterior fundus (AF) face patch, in the superior temporal sulcus, and anterior medial (AM) face patch, on the undersurface of the temporal lobe, in macaques presented with audiovisual, visual-only, and auditory-only renditions of natural movies of macaques vocalizing. The results revealed that 76% of neurons in face patch AF were significantly influenced by the auditory component of the movie, most often through enhancement of visual responses but sometimes in response to the auditory stimulus alone. By contrast, few neurons in face patch AM exhibited significant auditory responses or modulation. Control experiments in AF used an animated macaque avatar to demonstrate, first, that the structural elements of the face were often essential for audiovisual modulation and, second, that the temporal modulation of the acoustic stimulus was more important than its frequency spectrum. Together, these results identify a striking contrast between two face patches and specifically identify AF as playing a potential role in the integration of audiovisual cues during natural modes of social communication.

Keywords: audition; electrophysiology; face patches; multisensory integration; primate; vision.

Graphical abstract

Figure 1

Localization of recording sites in the STS and IT cortex (A and B) Functional overlays of AF and AM from monkey SP and monkey W, respectively, of an fMRI contrast of faces versus objects. The tract of the electrode is indicated with the red arrow targeted to the desired areas of recording. (C) Pant-threat vocalization from an unfamiliar macaque. (D) The same vocalization as performed by the avatar, both including the 500-ms still frame indicated by the frames labeled 1. (E) Presentation timeline of stimulus indicating the onset of the still frame indicated in red and the onset of the movie in green as well as the auditory stimulus, including a spectrogram.

Figure 2

Example responses from AF and AM The dark gray panel indicates the static frame although the light gray indicates the audiovisual movie stimulus (magenta), silent movie (blue), or vocalization (red). (A and B) Typical enhancement of AF neuron’s response for two different stimuli (two-way ANOVA; A, p Vis < 0.0001, p Aud = 0.3401, p Int < 0.0001; B, p Vis < 0.0001, p Aud = 0.8686, p Int < 0.0001). The horizontal black line within the rasters delineates the different recording sessions for the presented neurons. (C) Typical AM neuron’s response with little or no auditory modulation (p Vis < 0.0001; p Aud = 0.8014; p Int = 0.2002). (D–G) Additional example AF neuron responses. (D) Another typical AF non-linear multisensory enhanced response (p Vis < 0.0001; p Aud = 0.7110; p Int < 0.001). (E–G) Different profiles of audiovisual integration also expressed by neurons in AF. (E) A cell with a non-linear suppression of spiking in response to the audiovisual condition compared to the visual-only condition (p Vis < 0.0001; p Aud = 0.0086; p Int = 0.001). (F and G) Bimodal responses, where the response to the audiovisual movie mirrored the response to a unimodal condition (visual in F, p Vis = 0.0396, p Aud < 0.0001, p Int = 0.05, and auditory in G, p Vis = 0.7750, p Aud < 0.0001, p Int = 0.2888) along with a response to the other unimodal stimulus. Response types were determined by two-way ANOVA considering the presence or absence of the audio and visual stimulus components and their interaction. The index of modulation is shown in the corner of each spike density function. See also Figure S1.

Figure 3

Comparison of population responses to audiovisual stimuli of AF and AM neurons (A and C) Schematic representations of the relative positions of all the face patches specifically marking AF (A) and AM (C). (B and D) Plot of the proportions of neurons with significant modulation to each modality or the combination of modalities as calculated by two-way ANOVA for all stimuli (top row) and each stimulus analyzed independently (lower rows; AF, n = 119; AM, n = 55). (E) A scatterplot comparing the initial response the appearance of the still frame to the index of modulation for both AF and AM neurons. (F) Distribution of the mean index of modulation for each neuron for AF (blue) and AM (orange); the black line marks 0, the dashed blue line indicates the median of the AF distribution (0.1290), and the dashed red line indicates the median for the AM distribution (−0.0150). (G) Distribution of neurons for which a given number of stimuli demonstrate auditory modulation. See also Figures S2 and S3.

Figure 4

Responses to visual control stimuli (A and B) Single-cell example of responses to the different versions of the agonistic call stimulus. (A) portrays the average responses to the avatar producing an agonistic call although (B) shows the response of the same cell to the expanding disk stimulus matched to the same vocalization. (C and D) Population response of AF to audiovisual avatar stimuli comparing (C) the selectivity of cell responses to the audiovisual avatar stimuli to (D) the selectivity of cell responses to the audiovisual expanding disk control stimuli.

Figure 5

Responses to acoustic control stimuli (A and B) A single-cell example of responses to the different versions of the bark stimulus with (A) the average response of a single cell to the avatar bark stimulus and (B) the response to the avatar when a temporally modulated broadband noise (BBN) stimulus replaced the bark vocalization. (C) The distribution of the index of modulation for all calls across the population for both the avatar audiovisual stimuli and the avatar BBN control stimuli. (D and E) A single-cell example of the response to different versions of the agonistic call with (D) the cell response to the avatar agonistic stimulus and (E) the response to the avatar agonistic BBN stimulus. (F) The distribution of index of modulation to the tonal coos and agonistic calls.