Putting lexical constraints in context into the visual-world paradigm

Abstract

Prior eye-tracking studies of spoken sentence comprehension have found that the presence of two potential referents, e.g., two frogs, can guide listeners toward a Modifier interpretation of Put the frog on the napkin... despite strong lexical biases associated with Put that support a Goal interpretation of the temporary ambiguity (Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M. & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632-1634; Trueswell, J. C., Sekerina, I., Hill, N. M. & Logrip, M. L. (1999). The kindergarten-path effect: Studying on-line sentence processing in young children. Cognition, 73, 89-134). This pattern is not expected under constraint-based parsing theories: cue conflict between the lexical evidence (which supports the Goal analysis) and the visuo-contextual evidence (which supports the Modifier analysis) should result in uncertainty about the intended analysis and partial consideration of the Goal analysis. We reexamined these put studies (Experiment 1) by introducing a response time-constraint and a spatial contrast between competing referents (a frog on a napkin vs. a frog in a bowl). If listeners immediately interpret on the... as the start of a restrictive modifier, then their eye movements should rapidly converge on the intended referent (the frog on something). However, listeners showed this pattern only when the phrase was unambiguously a Modifier (Put the frog that's on the...). Syntactically ambiguous trials resulted in transient consideration of the Competitor animal (the frog in something). A reading study was also run on the same individuals (Experiment 2) and performance was compared between the two experiments. Those individuals who relied heavily on lexical biases to resolve a complement ambiguity in reading (The man heard/realized the story had been...) showed increased sensitivity to both lexical and contextual constraints in the put-task; i.e., increased consideration of the Goal analysis in 1-Referent Scenes, but also adeptness at using spatial constraints of prepositions (in vs. on) to restrict referential alternatives in 2-Referent Scenes. These findings cross-validate visual world and reading methods and support multiple-constraint theories of sentence processing in which individuals differ in their sensitivity to lexical contingencies.

Figure 18 (Appendix 1)

Proportion of looks to the Target and Competitor animal over time for those trials on which the listener happened to be fixating the Target animal at the onset of the first preposition. The data is plotted separately for 2-Referent Ambiguous and 2-Referent Unambiguous trials.

Figure 19 (Appendix 1)

Proportion of looks to the Target and Competitor animal over time for those trials on which the listener happened to be fixating the Competitor animal at the onset of the first preposition. The data is plotted separately for 2-Referent Ambiguous and 2-Referent Unambiguous trials.

Figure 20 (Appendix 1)

Proportion of looks to the Target and Competitor animal over time for those trials on which the listener happened to be fixating the Central Fixation Point (the smiley face) at the onset of the first preposition. The data is plotted separately for 2-Referent Ambiguous and 2-Referent Unambiguous trials.

Figure 1

(A) 1-Referent Scene type, which supports the Goal interpretation for on the napkin. These scenes contain, for instance, a Target animal (frog on a napkin), a Competitor animal (horse in a basket), an Incorrect Goal (an empty napkin), and a Correct Goal (a box). (B) 2-Referent Scene type, which supports the Modifier interpretation for on the napkin. These scenes contain the same props as 1-Referent Scenes except that a second frog replaces the horse as the Competitor animal.

Figure 2

Errors in hand actions for the put study across conditions. These rates are similar to those found for adults in Trueswell et al. (1999). Error bars reflect the 95% Confidence Interval.

Figure 3

Proportion of time spent looking at the Incorrect Goal (e.g., an empty napkin) from the onset of napkin until a hand action was completed. Main effects of Referential Context and Ambiguity were observed as well as an interaction between these two factors (i.e. 1-Referent Ambiguous contexts elicited the greatest proportion of looks to the Incorrect Goal; see text). Error bars reflect the 95% Confidence Interval.

Figure 4

Proportion of looks to the Incorrect Goal over time relative to the onset of napkin… across all four conditions.

Figure 5

(a) Target Advantage: Probability of fixating the Target animal (e.g., the frog on the napkin) instead of the Competitor animal (e.g., the frog in the basket) over time for 2-Referent Ambiguous and Unambiguous conditions. 0 ms on the x-axis marks the onset of the preposition on in the temporarily ambiguous phrase on the napkin. (b) Probability of fixating the Target animal and Competitor animal separately over time in 2-Referent Contexts.

Figure 6

(a) Proportion of looks to the Correct Goal over time in 2-Referent Scenes, relative to the onset of the second PP (e.g., into; marked by 0 ms on the x-axis), split by Ambiguity. (b) Proportion of looks to the Correct Goal over time, relative to the onset of into, split by Referential Scene and Ambiguity.

Figure 7

Proportion of looks to the Incorrect Goal over time as the Ambiguous phrase unfolds (relative to the onset of napkin), split by correct and incorrect action trials.

Figure 8

Significant positive correlation (r = .57; p < .0005; y = −.253 + .619*x) across conditions in the Visual-World task. The dependent variable is the Ambiguity Effect from 1-Referent Scenes (Proportion of Looks to the Incorrect Goal); the independent variable is the Ambiguity Effect from 2-Referent Scenes (Proportion of Looks to the Competitor animal). Correct actions only.

Figure 9

Reading time (ms) by condition. Main effects of Verb Type and Ambiguity were observed (ps < .01) as well as a reliable interaction between them (p < .05). Error bars reflect the 95% Confidence Interval.

Figure 10

Reading time (ms) by condition split by Type of Reader.

Figure 11

Proportion of errors involving the Incorrect Goal (Exp. 1) in each condition split by Type of Reader.

Figure 12

(a). Proportion of time spent looking to the Incorrect Goal during the first second after the onset of napkin, split by type of reader, including all trials (correct and incorrect actions). (b) Proportion of time spent looking to the Incorrect Goal during the first second after the onset of napkin, split by type of reader, correct action trials only.

Figure 13

(a) Proportion of looks to the Incorrect Goal over time as the Ambiguous phrase unfolds (relative to the onset of napkin), split by Type of Reader, including all trials (correct and incorrect actions). (b) Proportion of looks to the Incorrect Goal over time (relative to the onset of napkin), split by Type of Reader, correct action trials only.

Figure 14

(a) Probability of fixating the Target animal (e.g., the frog on the napkin) on all trials (i.e. correct and incorrect actions) instead of the Competitor animal (e.g., the frog in the basket) over time for 2-Referent Ambiguous and Unambiguous conditions, split by Type of Reader. 0 ms on the x-axis marks the onset of the preposition on in the temporarily ambiguous phrase on the napkin. (b) Correct trials only.

Figure 15

Proportion of looks to the Correct Goal over time in 2-Referent Scenes, relative to the onset of the second PP (e.g., into; marked by 0 ms on the x-axis), split by Type of Reader.

Figure 16

(a) Reading span performance by Type of Reader (ns); (b) Overall reading time by Type of Reader (ns); (c) Overall comprehension accuracy from Exp. 2 (ns); (d) Overall proportion of errors involving the Incorrect Goal, by Type of Reader (ns).

Figure 17

(a) Significant positive correlation (errors included; r = .52; p < .001; y = .236 + 2.11E-4*x) between the Ambiguity Effect from the Visual-World task in Exp. 1 (dependent variable; Proportion of Looks to the Incorrect Goal in 1-Referent Scenes, onset napkin to 1 sec thereafter) and the Ambiguity Effect in the Reading task in Exp. 2 (independent variable; residual Reading Times in ms). (b) Significant positive correlation (errors excluded); r = .38; p < .05; y = .163 + 1.61E-4*x) between the Ambiguity Effect in Exp. 1 (dependent variable) and the Ambiguity Effect in Exp. 2 (independent variable).