Intact speech-gesture integration in narrative recall by adults with moderate-severe traumatic brain injury

Abstract

Purpose: Real-world communication is situated in rich multimodal contexts, containing speech and gesture. Speakers often convey unique information in gesture that is not present in the speech signal (e.g., saying "He searched for a new recipe" while making a typing gesture). We examine the narrative retellings of participants with and without moderate-severe traumatic brain injury across three timepoints over two online Zoom sessions to investigate whether people with TBI can integrate information from co-occurring speech and gesture and if information from gesture persists across delays.

Methods: 60 participants with TBI and 60 non-injured peers watched videos of a narrator telling four short stories. On key details, the narrator produced complementary gestures that conveyed unique information. Participants retold the stories at three timepoints: immediately after, 20-min later, and one-week later. We examined the words participants used when retelling these key details, coding them as a Speech Match (e.g., "He searched for a new recipe"), a Gesture Match (e.g., "He searched for a new recipe online), or Other ("He looked for a new recipe"). We also examined whether participants produced representative gestures themselves when retelling these details.

Results: Despite recalling fewer story details, participants with TBI were as likely as non-injured peers to report information from gesture in their narrative retellings. All participants were more likely to report information from gesture and produce representative gestures themselves one-week later compared to immediately after hearing the story.

Conclusion: We demonstrated that speech-gesture integration is intact after TBI in narrative retellings. This finding has exciting implications for the utility of gesture to support comprehension and memory after TBI and expands our understanding of naturalistic multimodal language processing in this population.

Keywords: Gesture; Memory; Multimodal; Narrative recall; Speech-gesture integration; Traumatic brain injury.

Figure 1.

An example of a redundant (left) and complementary (right) gesture produced by the narrator during one of the Carl stories. In the redundant gesture example, the narrator says, “He formed the meat into balls,” while producing a meatball-patting movement. In the complementary gesture example, the narrator says, “He searched and searched for a new recipe,” while producing a typing movement.

Figure 2.

Mean number of story details recalled by NC and TBI participants at each of the three timepoints. Bars represent standard deviation of the mean. The analysis detected a significant main effect of group at the No Delay Timepoint such that participants with TBI were less likely to recall a story detail than NC peers. There were no significant interactions between participant group and delay, indicating that this pattern was similar at Short and Long Delay timepoints.

Figure 3.

Of the story details presented with complementary gestures that were recalled at each time point (max = 8), the proportion of participants’ retellings that matched the narrator’s speech (speech match), matched the narrator’s gesture (gesture match), or neither (other) for non-injured comparison participants (NC) and participants with traumatic brain injury (TBI).

Figure 4.

The percentage of the 89% and 100% credible intervals of posterior distributions of model parameters that fall within the ROPE limits (shown in transparent blue/purple). The percentage of the 89% CI for parameters are as follows: GroupTBI = 71.75%, DelaySHORT = 45.80%, DelayLONG = 5.89%, GroupTBI:DelaySHORT = 62.71%, GroupTBI:DelayLONG = 43.33%. All parameters have some degree of overlap between the 89% CI and ROPE region, providing inconclusive evidence for rejection or acceptance of the null hypotheses.

Figure 5.

Proportion of details in which participants produced no gesture, beat gestures, and representative gestures when retelling details in which the narrator gestured (n=16).