Neural dynamics of semantic composition

Abstract

Human speech comprehension is remarkable for its immediacy and rapidity. The listener interprets an incrementally delivered auditory input, millisecond by millisecond as it is heard, in terms of complex multilevel representations of relevant linguistic and nonlinguistic knowledge. Central to this process are the neural computations involved in semantic combination, whereby the meanings of words are combined into more complex representations, as in the combination of a verb and its following direct object (DO) noun (e.g., "eat the apple"). These combinatorial processes form the backbone for incremental interpretation, enabling listeners to integrate the meaning of each word as it is heard into their dynamic interpretation of the current utterance. Focusing on the verb-DO noun relationship in simple spoken sentences, we applied multivariate pattern analysis and computational semantic modeling to source-localized electro/magnetoencephalographic data to map out the specific representational constraints that are constructed as each word is heard, and to determine how these constraints guide the interpretation of subsequent words in the utterance. Comparing context-independent semantic models of the DO noun with contextually constrained noun models reflecting the semantic properties of the preceding verb, we found that only the contextually constrained model showed a significant fit to the brain data. Pattern-based measures of directed connectivity across the left hemisphere language network revealed a continuous information flow among temporal, inferior frontal, and inferior parietal regions, underpinning the verb's modification of the DO noun's activated semantics. These results provide a plausible neural substrate for seamless real-time incremental interpretation on the observed millisecond time scales.

Keywords: EEG/MEG; RSA; computational modelling; directed connectivity; speech.

Fig. 1.

Example of topic modeling results. Each verb (Upper) (e.g., “eat”) is represented as a distribution over 200 semantic topics (Middle), P(topic|verb), which reflects its semantic constraints over the DO noun. Each topic is a distribution over the vocabulary consisting of all the DO nouns from the large-scale corpora (Lower), P(DO noun|topic). Moreover, the meaning of a topic can be readily interpreted by the top words ranked by probability (e.g., topic 50 is a food topic).

Fig. 2.

Examples of verb (Left) and noun (Middle) topic vectors that separately capture verb semantic constraints and DO noun semantics. The verb-weighted noun topic vector (Right) models the meaning of the DO noun in the context of a prior verb by emphasizing topics that are preferred by the preceding verb through element-by-element multiplication between verb and noun topic vectors.

Fig. 3.

Illustration of the pipeline for ssRSA that correlates the dissimilarity generated by topic modeling (i.e., model RDM) and that encoded by brain activity (i.e., data RDM) using a spatiotemporal searchlight moving within a bilateral language mask at each time point during speech input. Model fits reflect when and where the information captured by the model is represented in the brain.

Fig. 4.

ssRSA results of model RDMs during the verb epoch (aligned to verb onset, extending 600 ms afterward to cover 1 SD of verb duration). (A) Verb topic RDM that captures verb semantics. (B) Verb topic entropy RDM modeling the strength of a verb’s semantic constraints. Significance was determined by 5,000 nonparametric permutations with vertex-wise P < 0.01 and cluster-wise P < 0.05. Horizontal orange bars indicate periods during which different model RDMs showed significant effects. Gray shading indicates the range of 1 SD for verb RP and verb offset.

Fig. 5.

ssRSA results of model RDMs during the noun epoch (aligned to noun onset, extending forward by 640 ms to cover 1 SD of noun duration). (A) Verb-weighted noun topic RDM that captures noun semantics as modified by the prior verb. (B) Noun topic RDM that models the context-independent semantics of the DO noun. (C) Verb–noun interaction RDM reflecting the interaction between verb and noun semantics. (D) Verb constraint error RDM measuring the ease with which the DO noun fits into the semantic constraints placed by the prior verb. Significance was determined by 5,000 nonparametric permutations with vertex-wise P < 0.01 and cluster-wise P < 0.05. Horizontal orange bars indicate significant periods for different model RDMs. The gray shading indicates the range of 1 SD for noun RP and noun offset.

Fig. 6.

Vertex-wise peak t value and significance duration of model RDMs during the verb epoch—verb topic RDM (A) and verb topic entropy RDM (B)—and during the noun epoch—verb-weight noun topic RDM (C), verb–noun interaction RDM (D), and verb constraint error RDM (E).

Fig. 7.

Directed connectivity analysis based on data RDMs constructed separately from two brain regions. (Left) The logic is that if region A has causal effects on region B, then the activity of A at a previous time point can be used to explain the current activity in B better than using only the previous activity of B alone, which is quantified by the partial correlation coefficients. (Right) The horizontal axis indicates the real time at which the speech unfolds, and the vertical axis indicates the time interval between the current time point and the previous time point used to calculate directed connectivity, thereby providing additional temporal information about the onset and duration of directed connectivity. t₀, current time point; dt, time interval between current and previous time points.

Fig. 8.

Directed connectivity results for (A) the L SMG/AG and LpMTG, showing significant model fit to the verb topic RDM during the verb epoch, and (B) the LIFG and LMTG, exhibiting significant model fit to the verb-weighted noun topic RDM during the noun epoch. Significance was determined by 5,000 nonparametric permutations with time point-wise P < 0.001 and cluster-wise P < 0.01. dt, time interval between the current time point and the previous time-point used to calculate directed connectivity; PCC, partial correlation coefficient.