Multimodality matters in numerical communication

Abstract

Modern society depends on numerical information, which must be communicated accurately and effectively. Numerical communication is accomplished in different modalities-speech, writing, sign, gesture, graphs, and in naturally occurring settings it almost always involves more than one modality at once. Yet the modalities of numerical communication are often studied in isolation. Here we argue that, to understand and improve numerical communication, we must take seriously this multimodality. We first discuss each modality on its own terms, identifying their commonalities and differences. We then argue that numerical communication is shaped critically by interactions among modalities. We boil down these interactions to four types: one modality can amplify the message of another; it can direct attention to content from another modality (e.g., using a gesture to guide attention to a relevant aspect of a graph); it can explain another modality (e.g., verbally explaining the meaning of an axis in a graph); and it can reinterpret a modality (e.g., framing an upwards-oriented trend as a bad outcome). We conclude by discussing how a focus on multimodality raises entirely new research questions about numerical communication.

Keywords: data visualization; gesture; graphs; multimodality; numerical cognition; numerical communication; quantifiers.

Figure 1

Multiple modalities to express exact numerals; compare Chrisomalis (2020, p. 151).

Figure 2

Visualization of temperature over time (1884–2021) for England, United Kingdom, using data from the United Kingdom Met Office. The use of color to convey numerical information encourages approximate rather than exact comparison (This #ShowYourStripes image by Ed Hawkins is licensed under CC BY 4.0. Available at: https://showyourstripes.info/).

Figure 3

Still image of Hans Rosling in the BBC Four The Joy of Stats pointing toward the y-axis; video available at: https://www.youtube.com/watch?v=jbkSRLYSojo (Accessed December 21, 2022). Reproduced with the permission of Wingspan Productions Ltd.

Figure 4

Final position of the sweeping gesture, which started by pointing to the origin and then swept along the x-axis. Reproduced with the permission of Wingspan Productions Ltd.

Figure 5

The lower-left quadrant is highlighted in three different ways: via speech (“down here”), a two-handed pointing gesture, and via tick marks becoming highlighted. Reproduced with the permission of Wingspan Productions Ltd.

Figure 6

Speech, gesture, and graphical display are combined to convey a conceptual mapping between quantity and area. Hans Rosling uses speech to explain how the size of each data point represents population size, while simultaneously forming a circle with two hands. An animated circle is superimposed on his hands, along with written text (“size=population”). Reproduced with the permission of Wingspan Productions Ltd.