Reevaluating pragmatic reasoning in language games

Abstract

The results of a highly influential study that tested the predictions of the Rational Speech Act (RSA) model suggest that (a) listeners use pragmatic reasoning in one-shot web-based referential communication games despite the artificial, highly constrained, and minimally interactive nature of the task, and (b) that RSA accurately captures this behavior. In this work, we reevaluate the contribution of the pragmatic reasoning formalized by RSA in explaining listener behavior by comparing RSA to a baseline literal listener model that is only driven by literal word meaning and the prior probability of referring to an object. Across three experiments we observe only modest evidence of pragmatic behavior in one-shot web-based language games, and only under very limited circumstances. We find that although RSA provides a strong fit to listener responses, it does not perform better than the baseline literal listener model. Our results suggest that while participants playing the role of the Speaker are informative in these one-shot web-based reference games, participants playing the role of the Listener only rarely take this Speaker behavior into account to reason about the intended referent. In addition, we show that RSA's fit is primarily due to a combination of non-pragmatic factors, perhaps the most surprising of which is that in the majority of conditions that are amenable to pragmatic reasoning, RSA (accurately) predicts that listeners will behave non-pragmatically. This leads us to conclude that RSA's strong overall correlation with human behavior in one-shot web-based language games does not reflect listener's pragmatic reasoning about informative speakers.

Fig 1. Example context type from F&G [20].

Fig 2. Examples of the visual context types from F&G.

The position of the speaker’s target was randomized, but appears on the left in the examples. Context Type indicates the number of objects that share the target’s features (here: #shape.#color). 2.2b is the critical pragmatically solvable context type (also shown in Fig 1). See text for details.

Fig 3. Example items for each of the visual context types used in Experiment 1.

The speaker’s target was always in the middle. The first column defines the context type in terms of the number of objects that have target’s shape (s) and color (c). The second column describes the feature overlap between competitors (for those contexts in which it affects RSA’s predictions). The fourth and fifth columns categorize the range of reasoning types in the Listener task, given a shape or color word respectively, defining 24 unique conditions (e.g., 1s2c + same shape + color word). Highlighted cells indicate the conditions for which the predictions from RSA and the baseline literal listener model differ: the pragmatically solvable conditions (green), and the pragmatically reducible conditions (blue). See text for more detail.

Fig 4. Experiment 2.

Example visual contexts for geometric and iconic stimulus types.

Fig 5. Experiment 2.

Regression tree results for the Listener task for the iconic stimuli only. Left: Color word condition. Right: Shape word condition. Each node shows the predicted number of responses (out of 10) and the percentage of observations in the node.

Fig 6. Key screens in Experiments 1 and 2 (left) and Experiment 3 (right).