Visual bodily signals as core devices for coordinating minds in interaction

Abstract

The view put forward here is that visual bodily signals play a core role in human communication and the coordination of minds. Critically, this role goes far beyond referential and propositional meaning. The human communication system that we consider to be the explanandum in the evolution of language thus is not spoken language. It is, instead, a deeply multimodal, multilayered, multifunctional system that developed-and survived-owing to the extraordinary flexibility and adaptability that it endows us with. Beyond their undisputed iconic power, visual bodily signals (manual and head gestures, facial expressions, gaze, torso movements) fundamentally contribute to key pragmatic processes in modern human communication. This contribution becomes particularly evident with a focus that includes non-iconic manual signals, non-manual signals and signal combinations. Such a focus also needs to consider meaning encoded not just via iconic mappings, since kinematic modulations and interaction-bound meaning are additional properties equipping the body with striking pragmatic capacities. Some of these capacities, or its precursors, may have already been present in the last common ancestor we share with the great apes and may qualify as early versions of the components constituting the hypothesized interaction engine. This article is part of the theme issue 'Revisiting the human 'interaction engine': comparative approaches to social action coordination'.

Keywords: coordination; gesture; language evolution; pragmatics; precursors; visual bodily signals.

Figure 1.

Example of an OHS (open hand supine, or palm-up-open-hand) gesture (a), and example of a pointing gesture directed at the addressee as a way of marking information constituting mutually shared knowledge which the speaker is referring to verbally (common ground pointing gesture (CG-point)) (b).

Figure 2.

Illustration of a multimodal repair sequence. T − 1, trouble source; T0, repair initiation; T + 1, repair solution [139]; G, gesture (G3a − G3d = gestural subcomponents). The speaker says (T − 1) ‘[he goes up another level and] sticks his head out of what looks like a a a skylight in the barn’ to describe a little boy who climbed up to the top of a barn looking out of a window in the roof. In response to the addressee initiating repair (T0) by asking ‘he does what?’, the original speaker says (T + 1) ‘he sticks his head out of what looks like a skylight’. Note that the speech at T − 1 and T + 1 differs marginally, the change consisting mainly of the last three words (in the barn) having been dropped and the utterance not being dysfluent. The accompanying gestures, however, do most of the repair work. The utterance at T − 1 includes two gestures. The first involves the flat hand, palm down, moving diagonally upwards and forwards depicting the boy's head moving up through the roof window. The second gesture is a pragmatic one and involves the hand, palm facing down and fingers spread, turning in the wrist (from left to right, 3×) which seems to convey uncertainty about whether the term ‘skylight’ is the best fit. Both gestures get revised as part of the utterance in T + 1: the first gesture now involves the hand, held in the same way as before, moving first to the speaker's own head before then moving upwards and forwards. This added movement clarifies that the upward moving hand is meant to depict a head moving up. The second gesture changes from a pragmatic to an iconic one, now outlining the square shape of the opening in the roof, presumably to illustrate its window-like features.

Figure 3.

Hypothetical timelines for the evolution of hallmarks of human communicative behaviour. Solid arrows, present; dotted lines, precursor present; question marks, domain of uncertainty. Note that the gesture types refer to existing descriptions and labels (iconic, metaphoric, beats: McNeill [34]; pragmatic: Kendon [68]; interactive: Bavelas et al. [111]). Reference to the interaction engine is in bold since this hypothesized construct entails the interactive and pragmatic skills listed above it.