What the eyes say about speaking

Abstract

To study the time course of sentence formulation, we monitored the eye movements of speakers as they described simple events. The similarity between speakers' initial eye movements and those of observers performing a nonverbal event-comprehension task suggested that response-relevant information was rapidly extracted from scenes, allowing speakers to select grammatical subjects based on comprehended events rather than salience. When speaking extemporaneously, speakers began fixating pictured elements less than a second before naming them within their descriptions, a finding consistent with incremental lexical encoding. Eye movements anticipated the order of mention despite changes in picture orientation, in who-did-what-to-whom, and in sentence structure. The results support Wundt's theory of sentence production.

Figure 1

Top panel shows a sample Active picture set, typically described with active sentences in all versions (The mouse is squirting the turtle with water and The turtle is squirting the mouse with water); bottom panel shows a sample Passive/Active picture set (The mailman is being chased by the dog and The mailman is chasing the dog). Within each set, the upper pictures are the Original and Role-Traded versions and the lower are their mirror-images.

Figure 2

One speaker’s eye movements over a pictured event. Successive fixations, time-stamped in ms from picture onset, show where the eye rested with the size of the circle indicating the length of fixation (starting at the location of the fixation point that preceded picture presentation). The word “girl” began 917 ms after the first fixation on the woman and the word “man” began 843 ms after the man was first fixated.

Figure 3

Changes in viewing across successive 4 ms intervals from picture onset during (a) Extemporaneous Speech, (b) Prepared Speech, (c) Patient Detection, and (d) Picture Inspection for all picture types and versions. Shown are the proportions, for each interval, of trials in which regions were fixated.

Figure 4

Changes in viewing across successive 50 ms intervals during extemporaneous speech for the major elements of Active events (left) and Passive/Active events (right) in their Original (top) and Role-traded (bottom) forms, collapsed over mirror-imaged versions. Time is shown relative to the onset of the head of the subject noun phrase.