Neural integration of speech and gesture in schizophrenia: evidence for differential processing of metaphoric gestures

Abstract

Gestures are an important component of interpersonal communication. Especially, complex multimodal communication is assumed to be disrupted in patients with schizophrenia. In healthy subjects, differential neural integration processes for gestures in the context of concrete [iconic (IC) gestures] and abstract sentence contents [metaphoric (MP) gestures] had been demonstrated. With this study we wanted to investigate neural integration processes for both gesture types in patients with schizophrenia. During functional magnetic resonance imaging-data acquisition, 16 patients with schizophrenia (P) and a healthy control group (C) were shown videos of an actor performing IC and MP gestures and associated sentences. An isolated gesture (G) and isolated sentence condition (S) were included to separate unimodal from bimodal effects at the neural level. During IC conditions (IC > G ∩ IC > S) we found increased activity in the left posterior middle temporal gyrus (pMTG) in both groups. Whereas in the control group the left pMTG and the inferior frontal gyrus (IFG) were activated for the MP conditions (MP > G ∩ MP > S), no significant activation was found for the identical contrast in patients. The interaction of group (P/C) and gesture condition (MP/IC) revealed activation in the bilateral hippocampus, the left middle/superior temporal and IFG. Activation of the pMTG for the IC condition in both groups indicates intact neural integration of IC gestures in schizophrenia. However, failure to activate the left pMTG and IFG for MP co-verbal gestures suggests a disturbed integration of gestures embedded in an abstract sentence context. This study provides new insight into the neural integration of co-verbal gestures in patients with schizophrenia.

Figure 1

Examples of the four experimental conditions [Straube et al. 2011a]. The stimulus material consisted of videos of an actor performing iconic (IC) and metaphoric (MP) co‐verbal gestures as well as two unimodal control conditions (spoken sentences without gestures [S] and gestures without sentences [G]). A screenshot of a typical video is shown for each condition. For illustrative purposes the spoken German sentences were translated into English and written in speech bubbles.

Figure 2

Activation patterns for each condition in contrast to baseline (fixation cross +) and in comparison to each other in patients with schizophrenia. Activation patterns for each of the four conditions in contrast to baseline (fixation cross), the bimodal conditions in contrast to the unimodal conditions (IC > S; IC > G; MP > S; MP > G) and the direct contrasts of the processing of iconic (IC) and metaphoric (MP) co‐verbal gestures (IC > MP; MP > IC; see Table I). IC: iconic co‐verbal gesture condition (concrete); MP: metaphoric co‐verbal gestures condition (abstract); S: speech without gesture; G: gesture without speech. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Figure 3

Group comparison of integration processes. (A) shows brain regions with increased activation for bimodal (IC: green; MP: red; IC ∩ MP: yellow) in contrast to unimodal processing of speech and gesture, as identified through conjunction analyses in the healthy control group. Green = iconic: [(IC > S ∩ IC > G) inclusively masked for S ∩ G], red = metaphoric: [(MP > S ∩ MP > G) inclusively masked for S ∩ G]. Yellow = overlap: [(IC > S ∩ IC > G) ∩ (MP > S ∩ MP > G) inclusively masked for S ∩ G]. (A) shows results from healthy subjects previously published [Straube et al. 2011a], now analyzed using SPM8. (B) shows brain regions with increased activation for bimodal (IC: green; MP: red; IC ∩ MP: yellow) in contrast to unimodal processing of speech and gesture, as identified through conjunction analyses in patients with schizophrenia. The contrasts and color coding are identical to (A). For patients with schizophrenia only the posterior temporal lobe is significantly activated for iconic co‐verbal gestures in contrast to the unimodal control conditions. (C) Overlap in conjunctional activation in patients (IC > S ∩ IC > G) and healthy subjects (IC > S ∩ IC > G) illustrated in light blue (for statistics see result section). For the MP condition no overlap was found. Bar graphs (D and E) illustrate contrast estimates for the left IFG and posterior MTG based on the extracted eigenvariate of the respective activation clusters. Error bars represent the standard errors of the mean. IC: iconic co‐verbal gesture condition (concrete); MP: metaphoric co‐verbal gestures condition (abstract); S: speech without gesture; G: gesture without speech; Pat: patient group; Con: control group. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Figure 4

Interaction of group (P, C) and gesture condition (IC, MP). This figure illustrates the significant interaction of group (P, C) and gesture condition (C‐MP > C‐IC) > (P‐MP > P‐IC). Bar graphs on the right represent the contrast estimates for activation of the left IFG (BA45) and hippocampus, which are representative for the whole activation pattern. Bar graphs are based on the extracted eigenvariate from predefined ROIs of the BA45 and the left Hippocampus (including all subregions). ROIs are created by the use of the Anatomy Toolbox (Eickhoff et al. 2005). Error bars represent the standard error of the mean. The opposite contrast revealed no significant activation (C‐MP > C‐IC) < (P‐MP > P‐IC). IC: iconic co‐verbal gesture condition (concrete); MP: metaphoric co‐verbal gestures condition (abstract); S: speech without gesture; G: gesture without speech; Pat: patient group; Con: control group. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Figure 5

Processing of IC and MP in patients in contrast to controls. This figure illustrate the additional processes, indicated by stronger activation for the processing of co‐verbal gestures, in the patient group compared with the healthy control group (P‐IC > C‐IC; P‐MP > C‐MP). For both gesture–speech conditions we found more activation in bilateral frontal and occipital structures (see Table III). The activation of the left IFG in both contrasts is more superior and posterior that the IFG activation of the interaction analyses (see Fig 4). Bar graphs at the right represent the contrast estimates for activation of the left IFG (BA44) and right IFG (BA45), which are representative for the whole activation pattern. Bar graphs are based on the extracted eigenvariate from predefined ROIs of the right BA45 and the left BA44. ROIs were created by means of the Anatomy Toolbox (Eickhoff et al. 2005). Error bars represent the standard errors of the mean. The opposite contrasts revealed no significant activation [(C‐MP > P‐MP) and (C‐IC > P‐IC)]. IC: iconic co‐verbal gesture condition (concrete); MP: metaphoric co‐verbal gestures condition (abstract); S: speech without gesture; G: gesture without speech; Pat: patient group; Con: control group. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]