TMS Reveals Dynamic Interaction between Inferior Frontal Gyrus and Posterior Middle Temporal Gyrus in Gesture-Speech Semantic Integration

Abstract

Semantic processing is an amodal process with modality-specific information integrated in supramodal "convergence zones" or "semantic hub" with executive mechanisms that tailor semantic representation in a task-appropriate way. One unsolved question is how frontal control region dynamically interacts with temporal representation region in semantic integration. The present study addressed this issue by using inhibitory double-pulse transcranial magnetic stimulation over the left inferior frontal gyrus (IFG) or left posterior middle temporal gyrus (pMTG) in one of eight 40 ms time windows (TWs) (3 TWs before and 5 TWs after the identification point of speech), when human participants (12 females, 14 males) were presented with semantically congruent or incongruent gesture-speech pairs but merely identified the gender of speech. We found a TW-selective disruption of gesture-speech integration, indexed by the semantic congruency effect (i.e., a cost of reaction time because of semantic conflict), when stimulating the left pMTG in TW1, TW2, and TW7 but when stimulating the left IFG in TW3 and TW6. Based on the timing relationship, we hypothesize a two-stage gesture-speech integration circuit with a pMTG-to-IFG sequential involvement in the prelexical stage for activating gesture semantics and top-down constraining the phonological processing of speech. In the postlexical stage, an IFG-to-pMTG feedback signal might be implicated for the control of goal-directed representations and multimodal semantic unification. Our findings provide new insights into the dynamic brain network of multimodal semantic processing by causally revealing the temporal dynamics of frontal control and temporal representation regions.SIGNIFICANCE STATEMENT Previous research has identified differential functions of left inferior frontal gyrus (IFG) and posterior middle temporal gyrus (pMTG) in semantic control and semantic representation, respectively, and a causal contribution of both regions in gesture-speech integration. However, it remains largely unclear how the two regions dynamically interact in semantic processing. By using double-pulse transcranial magnetic stimulation to disrupt regional activity at specific time, this study for the first time revealed critical time windows when the two areas were causally involved in integrating gesture and speech semantics. Findings suggest a pMTG-IFG-pMTG neurocircuit loop in gesture-speech integration, which deepens current knowledge and inspires future investigation of the temporal dynamics and cognitive processes of the amodal semantic network.

Keywords: TMS; gesture; inferior frontal gyrus; posterior middle temporal gyrus; semantic integration; speech.

Figure 1.

Stimuli and experimental design. A, Illustration of the 8 TWs (duration = 40 ms) relative to the IP of speech in which TMS was applied. Speech onset coincided with the DP of the gesture. Top, Example of a speech IP >120 ms. Bottom, Example of a speech IP ≤120 ms. B, A total of 1280 gesture-speech pairs were split into 24 blocks that were completed on 4 different days with 5-7 d apart between each other. There were 8 blocks for each stimulation site, and the order of blocks was counterbalanced using a Latin square across participants. In each block, double-pulse TMS was conducted on one stimulation site [the left IFG (red spot), the left pMTG (blue spot), or the vertex (black spot)] in all 8 TWs randomly. In each trial, a fixation cross was first presented on the center of the screen for 0.5-1.5 s, followed by a video of gesture and speech, and participants were asked to respond to the gender of the voice within 2 s. A feedback was given only if the response was incorrect.

Figure 2.

RTs of four experimental conditions (bar graphs) and the semantic congruent and incongruent conditions (line graphs) for the three stimulation sites. Sc, Semantically congruent; Si, semantically incongruent; Gc, gender congruent; Gi, gender incongruent. Rt_mean represents the mean RT for the Sc and Si conditions. Error bars in the bar graphs and shadows in the line graphs indicate SEM.

Figure 3.

TMS effects on the semantic congruency effect (A) and the gender congruency effect (B). TMS effect was defined as active-TMS minus vertex-TMS. The semantic congruency effect was calculated as the RT difference between semantically incongruent and semantically congruent pairs, whereas the gender congruency effect was calculated as the RT difference between gender incongruent and gender congruent pairs. *p < 0.05; **p < 0.01; one-sample t tests after FDR correction. Error bars represent SEM.

Figure 4.

TMS effects on semantically congruent and incongruent conditions. TMS effects, defined as active-TMS minus vertex-TMS, on RTs of the semantically congruent (red) and semantically incongruent (black) conditions are shown for stimulating the left pMTG and left IFG in 8 TWs. *p < 0.05; **p < 0.01; one-sample t tests after FDR correction. Shadows represent SEM.

Figure 5.

Summary of results and the proposed two-stage integration circuit in semantic processing of gestures and speech. A, Summary of critical time periods that the left IFG (red spot) and left pMTG (blue spot) get involved in gesture-speech semantic integration. B, A gesture-speech semantic integration circuit was posited in which there were a pMTG-to-IFG sequential involvement at the prelexical stage (before speech reached its IP) for accessing and activating gesture semantics and top-down modulation of phonological processing in the STG/STS, and an IFG-to-pMTG timing relationship at the postlexical stage (after speech IP) for controlled retrieval and multimodal semantic unification.