The Large-Scale Organization of Gestures and Words in the Middle Temporal Gyrus

Abstract

The middle temporal gyrus (MTG) has been shown to be recruited during the processing of words, but also during the observation of actions. Here we investigated how information related to words and gestures is organized along the MTG. To this aim, we measured the BOLD response in the MTG to video clips of gestures and spoken words in 17 healthy human adults (male and female). Gestures consisted of videos of an actress performing object-use pantomimes (iconic representations of object-directed actions; e.g., playing guitar), emblems (conventional gestures, e.g., thumb up), and meaningless gestures. Word stimuli (verbs, nouns) consisted of video clips of the same actress pronouncing words. We found a stronger response to meaningful compared with meaningless gestures along the whole left and large portions of the right MTG. Importantly, we observed a gradient, with posterior regions responding more strongly to gestures (pantomimes and emblems) than words and anterior regions showing a stronger response to words than gestures. In an intermediate region in the left hemisphere, the response was significantly higher to words and emblems (i.e., items with a greater arbitrariness of the sign-to-meaning mapping) than to pantomimes. These results show that the large-scale organization of information in the MTG is driven by the input modality and may also reflect the arbitrariness of the relationship between sign and meaning.SIGNIFICANCE STATEMENT Here we investigated the organizing principle of information in the middle temporal gyrus, taking into consideration the input-modality and the arbitrariness of the relationship between a sign and its meaning. We compared the middle temporal gyrus response during the processing of pantomimes, emblems, and spoken words. We found that posterior regions responded more strongly to pantomimes and emblems than to words, whereas anterior regions responded more strongly to words than to pantomimes and emblems. In an intermediate region, only in the left hemisphere, words and emblems evoked a stronger response than pantomimes. Our results identify two organizing principles of neural representation: the modality of communication (gestural or verbal) and the (arbitrariness of the) relationship between sign and meanings.

Keywords: emblems; gestures; organization; pantomimes; speech.

Figure 1.

Stimulus material and design. A–D, Still frames from video clips showing pantomimes (A; playing violin), emblems (B; thumb up), meaningless gestures (C), and spoken words (D). E, Experimental design and example trial. The experiment consisted of two parts (1: Gestures experiment; 2: Words experiment), using a block design. In the Gestures experiment, the conditions pantomimes, emblems (event and state), and meaningless gestures were presented. In the Words experiment, the conditions verbs (concrete and abstract) and nouns (concrete and abstract) were presented.

Figure 2.

Top, The MTG ROI was defined as the cortical region inferior to the superior temporal sulcus (STS) and superior to the inferior temporal sulcus (ITS). Bottom, Procedure used for the MTG ROI analysis.

Figure 3.

Sensitivity to meaningful compared with meaningless gestures. A, Results of the whole-brain RFX GLM contrast meaningful versus meaningless gestures. B, BOLD amplitude (expressed as β estimates) to meaningful (black) and meaningless (blue) gestures as a function of the y position along the left (left column) and right (right column) MTG. C, BOLD amplitude for pantomimes and meaningless gestures. D, BOLD amplitude for emblems and meaningless gestures. Gray bars represent clusters that survived correction for multiple comparisons using TFCE (for details, see Materials and Methods).

Figure 4.

Effects of different categories of meaningful stimuli. A, B, Results of the whole-brain contrasts Gestures versus Baseline (A) and Words versus Baseline (B). C, Beta estimates for pantomimes (red), emblems (green), and words (yellow) as a function of the y coordinate along the left (left) and right (right) MTG. Gray bars represent clusters that survived corrections for multiple comparisons using TFCE (for details, see Materials and Methods) for each of the pairwise comparisons; light gray represents pantomimes versus emblems; medium gray represents emblems versus words; dark gray represents pantomimes versus words.

Figure 5.

Comparison of abstract and concrete words. A, Comparison of the BOLD amplitude for concrete and abstract verbs in the left (left) and right (right) hemisphere. B, Comparison of the BOLD amplitude for concrete and abstract nouns in the left (left) and right (right) hemisphere. Gray bars represent clusters that survived corrections for multiple comparisons using TFCE (for details, see Materials and Methods).