The effect of the visual context in the recognition of symbolic gestures

Abstract

Background: To investigate, by means of fMRI, the influence of the visual environment in the process of symbolic gesture recognition. Emblems are semiotic gestures that use movements or hand postures to symbolically encode and communicate meaning, independently of language. They often require contextual information to be correctly understood. Until now, observation of symbolic gestures was studied against a blank background where the meaning and intentionality of the gesture was not fulfilled.

Methodology/principal findings: Normal subjects were scanned while observing short videos of an individual performing symbolic gesture with or without the corresponding visual context and the context scenes without gestures. The comparison between gestures regardless of the context demonstrated increased activity in the inferior frontal gyrus, the superior parietal cortex and the temporoparietal junction in the right hemisphere and the precuneus and posterior cingulate bilaterally, while the comparison between context and gestures alone did not recruit any of these regions.

Conclusions/significance: These areas seem to be crucial for the inference of intentions in symbolic gestures observed in their natural context and represent an interrelated network formed by components of the putative human neuron mirror system as well as the mentalizing system.

Figure 1. Experimental conditions.

Image frames captured from some of the videos presented. Examples of one gesture (asking for the bill in a bar) in the three types of presentation: isolated, within the context and the context without the gesture (respectively from left to right).

Figure 2. Subtraction analysis.

Signal increase for (a) G-CONTEXT – GESTURE; (b) CONTEXT – GESTURE and (c) G-CONTEXT – GESTURE. The coordinates of the planes shown are z = 53 (top axial planes); z = 18 (bottom axial planes); x = −43 (top sagital planes); x = 43 (botton sagital planes). The most relevant regions are labelled. (d) BOLD signal for the three conditions and both hemispheres. In Black: GESTURE, Orange: G-CONTEXT and Blue: CONTEXT. up) Left hemisphere; down) Right hemisphere.

Figure 3. Conjunction analysis.

Regions where G-CONTEXT was greater than GESTURE and CONTEXT.