The fractionation of spoken language understanding by measuring electrical and magnetic brain signals

Abstract

This paper focuses on what electrical and magnetic recordings of human brain activity reveal about spoken language understanding. Based on the high temporal resolution of these recordings, a fine-grained temporal profile of different aspects of spoken language comprehension can be obtained. Crucial aspects of speech comprehension are lexical access, selection and semantic integration. Results show that for words spoken in context, there is no 'magic moment' when lexical selection ends and semantic integration begins. Irrespective of whether words have early or late recognition points, semantic integration processing is initiated before words can be identified on the basis of the acoustic information alone. Moreover, for one particular event-related brain potential (ERP) component (the N400), equivalent impact of sentence- and discourse-semantic contexts is observed. This indicates that in comprehension, a spoken word is immediately evaluated relative to the widest interpretive domain available. In addition, this happens very quickly. Findings are discussed that show that often an unfolding word can be mapped onto discourse-level representations well before the end of the word. Overall, the time course of the ERP effects is compatible with the view that the different information types (lexical, syntactic, phonological, pragmatic) are processed in parallel and influence the interpretation process incrementally, that is as soon as the relevant pieces of information are available. This is referred to as the immediacy principle.

Figure 1

Modulation of the N400 amplitude as a result of a manipulation of the semantic fit between a lexical item and its sentence context. The grand average waveform is shown for electrode site Pz (parietal midline), for the best fitting word (high cloze; solid line), and a word that is less expected in the given sentence context (low cloze; dashed line). The sentences were visually presented word by word for every 600 ms. In the figure, the critical words (CWs) are preceded and followed by one word. The CW is presented at 600 ms on the time axis. Negativity is up. (Reprinted with permission from Hagoort & Brown (1994). Copyright Erlbaum.)

Figure 2

ERPs to visually presented syntactic prose sentences. These are sentences without a coherent semantic interpretation. A P600/SPS is elicited by a violation of the required number agreement between the subject noun phrase and the finite verb of the sentence. The averaged waveforms for the grammatically correct (solid line) and the grammatically incorrect (dashed line) words are shown for electrode site Pz (parietal midline). The word that renders the sentence ungrammatical is presented at 0 ms on the time axis. The waveforms show the ERPs to this and the following two words. Words were presented word by word, with an interval (stimulus onset asynchrony) of 600 ms. Negativity is plotted upwards; asterisk indicates the syntactic isolation. (Reprinted with permission from Hagoort & Brown (1994). Copyright Erlbaum.)

Figure 3

Connected speech. Distribution of the N200 and N400 effects for four left hemisphere sites (F3, FC3, C3 and P3), four midline sites (Fz, FCz, Cz and Pz) and four right hemisphere sites (F4, FC4, C4 and P4). The N200 effect was determined by subtracting the mean amplitude in the 150–250 ms latency window of the grand average ERP for the semantically congruent sentence-final words from the mean amplitudes of the grand average ERP for the semantically anomalous sentence-final words that did not share the same initial phonemes as the semantically congruous words. The N400 effect was determined in the same manner on the basis of the mean amplitudes in the 300–500 ms latency window (Van den Brink et al. 2001).

Figure 4

Connected speech. Difference waveforms from a representative electrode site (Cz), for semantically congruent versus semantically anomalous words with early RPs (dotted line; mean 229 ms) and late RPs (solid black line; mean 343 ms). The time axis is in milliseconds. Negative polarity is plotted upwards.

Figure 5

MEG recordings of the N400. The N400m component is observed from 300 to 600 ms. The topography of the evoked fields (planar gradients of the evoked fields) for (a) the congruent nouns and (b) the incongruent nouns in sentence context. For the incongruous nouns, an increased N400m is visible over the left hemisphere.

Figure 6

Grand average ERP waveform elicited by spoken singular nouns presented in a one-referent (solid line) and a two-referent (dotted line) context. The acoustic onset of the noun is at 0 ms, and negative polarity is plotted upwards. The waveforms are shown for a representative frontal midline site (Fz).

Figure 7

Discourse-dependent semantic anomaly effects by word position and length. Grand average ERPs, at Pz, elicited by spoken words that were coherent (solid line) or anomalous in discourse (dotted line), for words in (a) sentence-medial and (b) sentence-final position, as well as for (c) short words (below 550 ms and a mean duration of 451 ms) and (d) long words (over 550 ms and a mean duration of 652 ms). The horizontal bars mark the range of acoustic word offsets for short and long CWs. (Reprinted with permission from Van Berkum et al. (2003). Copyright Elsevier B. V.)