Systematic mappings between semantic categories and types of iconic representations in the manual modality: A normed database of silent gesture

Abstract

An unprecedented number of empirical studies have shown that iconic gestures-those that mimic the sensorimotor attributes of a referent-contribute significantly to language acquisition, perception, and processing. However, there has been a lack of normed studies describing generalizable principles in gesture production and in comprehension of the mappings of different types of iconic strategies (i.e., modes of representation; Müller, 2013). In Study 1 we elicited silent gestures in order to explore the implementation of different types of iconic representation (i.e., acting, representing, drawing, and personification) to express concepts across five semantic domains. In Study 2 we investigated the degree of meaning transparency (i.e., iconicity ratings) of the gestures elicited in Study 1. We found systematicity in the gestural forms of 109 concepts across all participants, with different types of iconicity aligning with specific semantic domains: Acting was favored for actions and manipulable objects, drawing for nonmanipulable objects, and personification for animate entities. Interpretation of gesture-meaning transparency was modulated by the interaction between mode of representation and semantic domain, with some couplings being more transparent than others: Acting yielded higher ratings for actions, representing for object-related concepts, personification for animate entities, and drawing for nonmanipulable entities. This study provides mapping principles that may extend to all forms of manual communication (gesture and sign). This database includes a list of the most systematic silent gestures in the group of participants, a notation of the form of each gesture based on four features (hand configuration, orientation, placement, and movement), each gesture's mode of representation, iconicity ratings, and professionally filmed videos that can be used for experimental and clinical endeavors.

Keywords: Iconicity; Modes of representation; Normed database; Perception of iconicity; Silent gesture.

Fig. 1

Examples of different modes of representation in silent gesture. “To smoke” implements the acting strategy, because the body reenacts the action of smoking. “To go down” implements the representing strategy, because two wiggling fingers depict two legs descending. “House” is depicted using the drawing strategy, by tracing a pentagon. In “bird” the gesturer uses the personification strategy, because the features of the referent are mapped onto his body

Fig. 2

Examples of gestural descriptions, following Bressem (2013). Each gesture is described according to its hand shape, orientation, movement, and placement. In some cases, gestures were produced with both hands (e.g., “to write”), and therefore a description is provided for both hands

Fig. 3

Examples of systematic and nonsystematic gestures produced for certain concepts. The form of the gesture “telephone” was very consistent across participants, so it was regarded as systematic and was analyzed further. For the concept “to break,” most participants produced the same gestural form, so it was also included for further analysis. There was high variability in the form of the gesture “to cook,” and given that ten people did not converge on the same structure, this concept was excluded from further analysis

Fig. 4

Proportions of gestures showing different types of modes of representation (acting, drawing, personification, and representing) per semantic category. Number of concepts per semantic category: actions with objects = 28; actions without objects = 21; animate = 10; manipulable objects = 34; nonmanipulable objects = 17

Fig. 5

Distribution of mean iconicity ratings for all gestures according to their mode of representation. The black dot marks the mean iconicity rating for each type of iconic strategy. The length of each violin represents the distribution of the data points along the iconicity scale. The width of the violin represents the concentration of data points around a specific value on the iconicity scale

Fig. 6

Distribution of mean iconicity ratings for all systematic gestures according to their semantic category. The black dot marks the mean iconicity rating for each mode of representation. The length of each violin represents the distribution of the data points along the iconicity scale. The width of the violin represents the concentration of data points around a specific value on the iconicity scale

Fig. 7

Box plot of mean iconicity ratings for each semantic category according to their mode of representation. Within the acting mode of representation, both types of actions (in light and dark blue) are at the upper end of the iconicity scale, and both types of objects (in orange and yellow) are at the lower end. Within the representing strategy, actions with objects and manipulable objects lie at the upper end of the iconicity scale, and the other categories are at the lower values. Personification is almost only used for animate entities. The drawing strategy is used primarily for both types of objects (mainly nonmanipulable objects), and they have low iconicity ratings