Neural Basis Of Sound-Symbolic Pseudoword-Shape Correspondences

Abstract

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like 'mohloh' and 'kehteh', are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants ( n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword ('mohloh' or 'kehteh') and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.

Highlights: fMRI investigation of sound-symbolic correspondences between auditory pseudowords and visual shapesFaster reaction times for congruent than incongruent audiovisual stimuliGreater activation in auditory and visual cortices for congruent stimuliHigher classification accuracy for congruent stimuli in language and visual areasSound symbolism involves language processing and multisensory integration.

Figure 1.

Task design and classification analysis scheme. (A) There were four possible stimuli, reflecting unique combinations of two visual shapes, pointed or rounded, and two auditory pseudowords, ‘kehteh’ or ‘mohloh’. (B) Pseudoword-shape pairs were presented simultaneously and participants pressed a response button indicating whether the pair were a ‘match’ (i.e., congruent) or a ‘mismatch’ (i.e., incongruent). (C) In the within-congruency classification analysis, leave-one-run-out cross-validation was used to test for patterns of brain activity discriminating the two congruent conditions or the two incongruent conditions. (D) in the between-congruency classification analyses, classifiers were trained to distinguish between the two conditions in which the visual shapes differed but the auditory pseudoword was the same (e.g., ‘kehteh’/pointed shape vs. ‘kehteh’/rounded shape). The classifier was then tested on the two remaining conditions containing the other pseudoword (e.g., ‘mohloh’/pointed shape vs. ‘mohloh’/rounded shape). Similarly, we trained classifiers to distinguish between auditory pseudowords holding the shape constant (e.g., ‘kehteh’/pointed shape vs. ‘mohloh’/pointed shape) and then tested them on the other shape (e.g., ‘kehteh’/rounded shape vs. ‘mohloh’/rounded shape). This approach differentiates whether patterns of brain activity were specific to features of the visual/auditory stimulus, or to the congruency/incongruency of the condition.

Figure 2.

Univariate results. Whole-brain map showing significant effects from the incongruent vs. congruent contrasts (topological FDR-corrected; activation clusters with cluster-forming threshold of p < 0.001 and FDR cluster-level correction of q < 0.05).

Figure 3.

Whole-brain searchlight maps showing significant above-chance within-congruency classification of the two congruent conditions (congruent decoding) and the two incongruent conditions (incongruent decoding). All maps whole-brain topological FDR-corrected; activation clusters with cluster-forming threshold of p < 0.001 and FDR cluster-level correction of q < 0.05).

Figure 4.

Whole-brain searchlight map showing significantly higher classification of the congruent conditions relative to the incongruent conditions and mean decoding accuracy in (A) left inferior frontal gyrus, pars opercularis (Broca’s area: L IFGpo), (B) left supramarginal gyrus (L SMG), and (C) right mid-occipital gyrus (MOG). Mean decoding accuracy was extracted from the peak voxel coordinates for each of the significant clusters in A-C. No region showed significantly higher above-chance classification of the incongruent conditions. All maps whole-brain topological FDR-corrected; activation clusters with cluster-forming threshold of p < 0.001 and FDR cluster-level correction of q < 0.05); error bars show ± standard error of the mean.

Figure 5.

Between-congruency classification. (A) Whole-brain searchlight maps showing significant above-chance decoding of either the visual shape or the congruency of the stimuli. (B) Whole-brain searchlight maps showing significant above-chance decoding of either the auditory pseudoword or the congruency of the stimuli. All maps whole-brain topological FDR-corrected; activation clusters with cluster-forming threshold of p < 0.001 and FDR cluster-level correction of q < 0.05).