Seeing the Unexpected: How Brains Read Communicative Intent through Kinematics

Abstract

Social interaction requires us to recognize subtle cues in behavior, such as kinematic differences in actions and gestures produced with different social intentions. Neuroscientific studies indicate that the putative mirror neuron system (pMNS) in the premotor cortex and mentalizing system (MS) in the medial prefrontal cortex support inferences about contextually unusual actions. However, little is known regarding the brain dynamics of these systems when viewing communicatively exaggerated kinematics. In an event-related functional magnetic resonance imaging experiment, 28 participants viewed stick-light videos of pantomime gestures, recorded in a previous study, which contained varying degrees of communicative exaggeration. Participants made either social or nonsocial classifications of the videos. Using participant responses and pantomime kinematics, we modeled the probability of each video being classified as communicative. Interregion connectivity and activity were modulated by kinematic exaggeration, depending on the task. In the Social Task, communicativeness of the gesture increased activation of several pMNS and MS regions and modulated top-down coupling from the MS to the pMNS, but engagement of the pMNS and MS was not found in the nonsocial task. Our results suggest that expectation violations can be a key cue for inferring communicative intention, extending previous findings from wholly unexpected actions to more subtle social signaling.

Keywords: communication; connectivity; fMRI; gesture; intention.

Figure 1

Still frames of a stick-light figure and a comparison with the corresponding video images. The lower panel depicts a series of still frames from one of the videos recorded in Trujillo et al. (2018) at various stages of action completion. The upper panel depicts the corresponding stick-light figure derived from the kinematics of this action. Note that the images in the upper panel represent what was seen by participants, who had no exposure to the video images. Figure was adapted with permission from Trujillo et al. (2019).

Figure 2

Overview of trial progression. The upper panel depicts the Social Task, while the lower panel depicts the Nonsocial Handedness Task. Participants first saw a single prime word, followed by a fixation cross of variable length, then the video, and, finally, the task-specific response screen.

Figure 3

Overview of GLM results. The top panels (A, C) depict slices from the Social Task, while the bottom panels (B, D) depict the Handedness Task. Red areas indicate a significant (P < 0.001) correlation between BOLD response and video communicativeness. The red color bars show the corresponding T values. Panels A and B provide a slice-by-slice overview of the 2 tasks, while panels C and D provide a 3D rendering of the same data, with significant areas of interest highlighted (IFG = inferior frontal gyrus; MFG = middle frontal gyrus).

Figure 4

Overview of winning DCM models. (A) The winning model for the Social Task. (B) The exceedance probability. In all models, circles depict the individual regions, while arrows depict the intrinsic, directional coupling between them. Video viewing is modeled as a driving input to the regions, while communicativeness is modeled as a modulator of coupling strength. (C) The 2 high-probability models for the Handedness Task. (D) The exceedance probabilities for these models.