Expectancy effects in the EEG during joint and spontaneous word-by-word sentence production in German

Abstract

Our aim in the present study is to measure neural correlates during spontaneous interactive sentence production. We present a novel approach using the word-by-word technique from improvisational theatre, in which two speakers jointly produce one sentence. This paradigm allows the assessment of behavioural aspects, such as turn-times, and electrophysiological responses, such as event-related-potentials (ERPs). Twenty-five participants constructed a cued but spontaneous four-word German sentence together with a confederate, taking turns for each word of the sentence. In 30% of the trials, the confederate uttered an unexpected gender-marked article. To complete the sentence in a meaningful way, the participant had to detect the violation and retrieve and utter a new fitting response. We found significant increases in response times after unexpected words and - despite allowing unscripted language production and naturally varying speech material - successfully detected significant N400 and P600 ERP effects for the unexpected word. The N400 EEG activity further significantly predicted the response time of the subsequent turn. Our results show that combining behavioural and neuroscientific measures of verbal interactions while retaining sufficient experimental control is possible, and that this combination provides promising insights into the mechanisms of spontaneous spoken dialogue.

Figure 1

Paradigm overview: setup, picture-naming task screen state, and word-by-word task and timeline. (a) Setup: Participant with EEG cap sat on the left hand side in front of a computer screen (no study participant is shown). A microphone was placed near her mouth. The confederate sat next to her in front of a second computer screen and a second microphone, as well as a table with keyboard and mouse. (b) Picture-naming task: As soon as the participant named the presented picture, the confederate clicked on the respective synonym or typed in a third option. Namings were automatically saved for the experiment. (c) Word-by-word timeline: Screen states and streams of participant and confederate. Example of an unexpected trial: participant named the picture ‘das Sofa’, expects the article ‘das’ and hears the article ‘die’. Steady presentation times of fixation cross and words and picture are given in seconds. During the interactive production part (lilac box), the participant saw a steady white fixation cross. The confederate started the sentence with word 1 (sentence subject), the participant then uttered word 2 (sentence verb), the confederate continued with word 3 (sentence article) and the participant ended the sentence with word 4 (sentence object). The interactive production part ended with the button press of the confederate. A blank screen was presented between trials. Average word duration for each word of a sentence, as well as turn-times from word to word are given in seconds. The ERP is computed for the expected/unexpected article (i.e., time-locked to the onset of word 3). The response time is contrasted between offset of word 3 until onset of word 4 after expected and unexpected articles. Sentence translation: “Tina sees the couch.” Cue verb is shown in the infinitive: ‘sehen’ – ‘to see’. Example sofa drawing not used in experiment (real-life object pictures were used).

Figure 2

Turn-times from word 3 (expected/unexpected article) to word 4 (object noun). (a) Average turn-times from offset of word 3 to onset of word 4 are shown in seconds for each participant for expected (blue) and unexpected (magenta) conditions plus standard deviation (black lines). Grand average turn-time (M) is shown below. (b) Predicted average turn-times from offset of word 3 to onset of word 4 from the calculated GLMM, shown for each participant (y-axis) for expected (blue) and unexpected (magenta) conditions. Grand average predicted turn-time for the expected condition is shown as blue vertical line and for unexpected conditions as magenta vertical line. (c) Average turn-times from offset of word 3 to onset of word 4 (y-axis; in seconds) are contrasted to interindividual frequencies of naming (x-axis; percentage bin center points connected by a line).

Figure 3

Grand average ERPs of word 3 shown for midline ROIs and quadrant ROIs for expected (blue) and unexpected (magenta) conditions. Same regions were used for statistical analysis. The N400 effect time window is highlighted in light grey (250–450 ms). The P600 time window is highlighted in dark grey (500–700 ms). Zero point is the onset of word 3, the expected or unexpected article. (a) Electrode map shows specific regions of all seven ROIs with highlighted Cz. (b) Grand average ERPs of word 3 averaged over left anterior quadrant. (c) Grand average ERPs of word 3 averaged over frontal midline. (d) Grand average ERPs of word 3 averaged over right anterior quadrant. (e) Grand average ERPs of word 3 averaged over central midline. (f) Grand average ERPs of word 3 averaged over left posterior quadrant. (g) Grand average ERPs of word 3 averaged over posterior midline. (h) Grand average ERPs of word 3 averaged over right posterior quadrant. (i) Difference grand average topographies for the specific time windows are shown for the N400 and P600 effect. (j) ERP plots axis description with mean word duration (of confederate) plus standard deviation below (same time scale). *p < 0.05, **p < 0.01, ***p < 0.001.