Convergent and invariant object representations for sight, sound, and touch

Abstract

We continuously perceive objects in the world through multiple sensory channels. In this study, we investigated the convergence of information from different sensory streams within the cerebral cortex. We presented volunteers with three common objects via three different modalities-sight, sound, and touch-and used multivariate pattern analysis of functional magnetic resonance imaging data to map the cortical regions containing information about the identity of the objects. We could reliably predict which of the three stimuli a subject had seen, heard, or touched from the pattern of neural activity in the corresponding early sensory cortices. Intramodal classification was also successful in large portions of the cerebral cortex beyond the primary areas, with multiple regions showing convergence of information from two or all three modalities. Using crossmodal classification, we also searched for brain regions that would represent objects in a similar fashion across different modalities of presentation. We trained a classifier to distinguish objects presented in one modality and then tested it on the same objects presented in a different modality. We detected audiovisual invariance in the right temporo-occipital junction, audiotactile invariance in the left postcentral gyrus and parietal operculum, and visuotactile invariance in the right postcentral and supramarginal gyri. Our maps of multisensory convergence and crossmodal generalization reveal the underlying organization of the association cortices, and may be related to the neural basis for mental concepts.

Keywords: auditory; concepts; crossmodal; fMRI; multisensory; multivariate pattern analysis; perception; tactile; visual.

Figure 1

Univariate analysis of brain regions activated by auditory, visual, and tactile stimuli. Hearing sounds most strongly activated the superior temporal lobe, peaking in Heschl's gyrus and the planum temporale. Watching (silent) videos broadly activated occipital and parietal cortices, peaking in the occipital pole. Touching objects resulted in an activation pattern that peaked in pre‐ and postcentral gyri and included additional, broad aspects of the parietal lobe. The simultaneous presentation of objects in all three modalities (AVT), as compared to rest, resulted in a broad cortical activation pattern which peaked in the auditory, visual, and somatosensory cortices, and resembled a concatenation of the unimodal maps. A “superadditive” map of regions in which simultaneous AVT presentation evoked greater activity than the sum of the separate unimodal activations revealed a multimodal potentiation effect. All slices are in radiological convention (right side of image corresponds to left side of brain). MNI coordinates specify the slices displayed, and not the locations of peak voxels. All volumetric statistical images in this article may be downloaded at the links provided in the text. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Figure 2

Intramodal searchlight analysis. Left, Group‐average classification accuracy for sounds, videos, and touches. Auditory stimuli were most accurately decoded in the auditory cortices, peaking along Heschl's gyrus. Visual stimuli were most accurately decoded in the calcarine sulcus and occipital pole. Tactile stimuli were most accurately decoded along the pre‐ and postcentral gyri. Successful classification, for each modality, also extended beyond the respective sensory cortices, into the association cortices as well as the early sensory cortices of the other modalities. Maps were thresholded at the P level of 0.01 (t > 2.567; DOF = 17) and then a cluster mass P level of 0.05, as determined by a nonparametric one‐sample t‐test (5,000 permutations of random sign‐flipping of the data). Right, Overlaps among the intramodal classifications. A crossmodal overlap score was calculated by summing the normalized accuracy values of the input maps and selecting only the shared voxels. Auditory and visual stimuli were jointly best decoded in parietal operculum, supramarginal gyrus, posterior temporal cortex, and temporal and occipital fusiform gyri. Auditory and tactile stimuli were jointly best decoded in planum temporale, parietal operculum, and supramarginal gyrus. Videos and touches were jointly best decoded in superior parietal lobule, supramarginal gyrus, parietal operculum, and lateral and inferior occipital cortex. Stimuli from all three modalities were jointly best decoded in the parietal operculum, supramarginal gyrus, and secondary somatosensory cortex, as well as lateral and ventral occipital cortices. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Figure 3

Crossmodal searchlight analysis. Classifiers trained to distinguish objects presented in one sensory modality were used to distinguish the same objects presented in a different sensory modality. Thus, above‐chance performance indicates the presence of modality‐invariant information about the identity of the object. Top, Audiovisual invariance was detected in a right hemisphere cluster located at the junction of lateral occipital cortex and posterior temporal cortex. Middle, Audiotactile invariance was detected in left secondary somatosensory cortex, along the inferior postcentral gyrus and parietal operculum. Bottom, Visuotactile invariance was detected in two right hemisphere clusters: one that extended from postcentral gyrus and superior parietal lobule to supramarginal gyrus and into parietal operculum, and the other in medial premotor cortex. Maps were thresholded at the P level of 0.01 (t > 2.567; DOF = 17) and then a cluster mass P level of 0.05, as determined by a nonparametric one‐sample t‐test (5,000 permutations of random sign‐flipping of the data). [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Figure 4

Overview of results. (A) Intramodal classifications and their overlaps (multimodal convergence). Red, green, and blue voxels indicate successful decoding of auditory, visual, and tactile stimuli, respectively. Arrows indicate the overlay of different maps. Overlapping regions are represented in additive hues. (B) Crossmodal classifications (multimodal invariance). The A–V (yellow), A–T (magenta), and V–T (cyan) invariant regions are a subset of the regions identified by the corresponding intramodal overlaps from A (presented using the same colors). The partially transparent central brain shows the overlay of the three types of invariance, with an inset brain showing the view angle. (C) A schematic view of modal and supramodal information processing. A stepwise convergence of sensory information occurs at border regions of the relevant sensory cortices. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]