A unified coding strategy for processing faces and voices

Abstract

Both faces and voices are rich in socially-relevant information, which humans are remarkably adept at extracting, including a person's identity, age, gender, affective state, personality, etc. Here, we review accumulating evidence from behavioral, neuropsychological, electrophysiological, and neuroimaging studies which suggest that the cognitive and neural processing mechanisms engaged by perceiving faces or voices are highly similar, despite the very different nature of their sensory input. The similarity between the two mechanisms likely facilitates the multi-modal integration of facial and vocal information during everyday social interactions. These findings emphasize a parsimonious principle of cerebral organization, where similar computational problems in different modalities are solved using similar solutions.

Figure 1

Face and voice-selective neural responses. (A) Left: face-selective areas revealed with functional MRI (fMRI) are shown in the occipital temporal cortex. Right: the voice-selective areas are found in superior temporal sulcus and gyrus. (B) Left: faces elicit greater event related potential (ERP) amplitudes than non-faces 170 ms after stimulus onset – N170 in occipito-temporal electrodes (red line – faces). Right: voices elicit greater amplitudes that non-voice sounds 200 ms after stimulus onset in fronto-temporal electrodes (red line – voices). Reproduced, with permission, from . (C) Left: transcranial magnetic stimulation (TMS) to the occipital face area selectively disrupts face but not body discrimination. Adapted from . Right: TMS to the temporal voice area selectively disrupts voice/nonvoice discrimination. Reproduced, with permission, from . (D) Left: face-selective areas found in the superior temporal sulcus of the macaque brain. Reproduced, with permission, from . Right: voice-selective areas were found in the superior temporal plane of the macaque brain. Reproduced, with permission, from .

Figure I

Perceptual aftereffects of ‘anti-face’ and ‘anti-voice’ adaptation. (A–C) Anti-face adaptation. (A) Four face identities used in a recognition task (left column) and their corresponding ‘anti-face’ versions (right column); note the very different identity precepts associated with a face and its anti-face; yet, they are related in that averaging them together results in the average face. (B) Stimuli used in recognition tasks represented in a theoretical multidimensional space centered on the average face (blue circle). Green circles indicate learned identities. Red circles indicate anti-faces. (C) Psychophysical labeling functions obtained as a function of increased identity strength at baseline (no adaptation: continuous line, open symbols) and after adaptation (closed symbols) with matched (continuous line) and non-matched (dashed line) anti-face adaptors. Note the greater aftereffects induced by matched anti-face adaptors and the strong identity percept associated with the otherwise identity neutral average face (identity strength 0) after adaptation with matched anti-faces. Reproduced, with permission, from . (D–E) Anti-voice adaptation. (D) Three voice stimuli (brief syllables represented by their spectrogram) shown in a theoretical multidimensional space, with an averaged voice in its center, and with their corresponding anti-voice stimuli (on the green circle). (E) Psychophysical labeling function obtained as a function of increased identity strength at baseline (no adaptation: orange symbols) and after adaptation with matched (blue symbols) and non-matched (pink symbols) anti-voice adaptors. Note, as for faces, the greater aftereffects induced by adaptation with matched anti-voice adaptors. Reproduced, with permission, from .

Figure I

Face and voice attractiveness judgments as a function of averaging. (A) Face composites generated by averaging 32 male faces (left) and 64 female faces (right). (B) Attractiveness ratings as a function of number of face averaged. Note the steady increase in attractiveness ratings with increasing number of averaged faces, for both male (left) and female (right) faces. Reproduced, with permission, from . (C) Spectrograms of voice composites generated by averaging an increasing number of voices of the same gender (different speakers uttering the syllable ‘had’). (D) Attractiveness ratings as a function of number of voices averaged. Note the steady increase in attractiveness ratings with increasing number of averaged voices, for both male (left) and female (right) voices. Reproduced, with permission, from .