Discourse-mediation of the mapping between language and the visual world: eye movements and mental representation

Abstract

Two experiments explored the mapping between language and mental representations of visual scenes. In both experiments, participants viewed, for example, a scene depicting a woman, a wine glass and bottle on the floor, an empty table, and various other objects. In Experiment 1, participants concurrently heard either 'The woman will put the glass on the table' or 'The woman is too lazy to put the glass on the table'. Subsequently, with the scene unchanged, participants heard that the woman 'will pick up the bottle, and pour the wine carefully into the glass.' Experiment 2 was identical except that the scene was removed before the onset of the spoken language. In both cases, eye movements after 'pour' (anticipating the glass) and at 'glass' reflected the language-determined position of the glass, as either on the floor, or moved onto the table, even though the concurrent (Experiment 1) or prior (Experiment 2) scene showed the glass in its unmoved position on the floor. Language-mediated eye movements thus reflect the real-time mapping of language onto dynamically updateable event-based representations of concurrently or previously seen objects (and their locations).

Fig. 1

Example scene from Experiments 1 and 2. See the main text for the accompanying sentential stimuli.

Fig. 2

Percentage of trials in Experiment 1 with fixations on the regions of interest corresponding to the table and the glass in the ‘moved’ and ‘unmoved’ conditions during ‘she will pick up the bottle and pour the wine carefully into the glass’ or its equivalent across trials. The percentages reflect the proportion of trials on which each of the regions of interest was fixated at each moment in time, and were calculated at each successive 25 ms from the synchronization point. See the main text for a description of the resynchronization process. The region of the graph corresponding to the target noun phrase ‘the glass’ is highlighted.

Fig. 3

Example regions of interest, shown in black, for Experiment 2 superimposed over an example scene.

Fig. 4

Percentage of trials in Experiment 2 with fixations on the regions of interest corresponding to where the table, glass, or distractor had been during ‘she will pick up the bottle and pour the wine carefully into the glass’ or its equivalent across trials. The percentages were calculated as for Experiment 1. Panel A shows the data from the ‘moved’ condition; Panel B shows the ‘unmoved’ data.