Co-speech gestures influence the magnitude and stability of articulatory movements: evidence for coupling-based enhancement

Abstract

Humans rarely speak without producing co-speech gestures of the hands, head, and other parts of the body. Co-speech gestures are also highly restricted in how they are timed with speech, typically synchronizing with prosodically-prominent syllables. What functional principles underlie this relationship? Here, we examine how the production of co-speech manual gestures influences spatiotemporal patterns of the oral articulators during speech production. We provide novel evidence that words uttered with accompanying co-speech gestures are produced with more extreme tongue and jaw displacement, and that presence of a co-speech gesture contributes to greater temporal stability of oral articulatory movements. This effect-which we term coupling enhancement-differs from stress-based hyperarticulation in that differences in articulatory magnitude are not vowel-specific in their patterning. Speech and gesture synergies therefore constitute an independent variable to consider when modeling the effects of prosodic prominence on articulatory patterns. Our results are consistent with work in language acquisition and speech-motor control suggesting that synchronizing speech to gesture can entrain acoustic prominence.

Keywords: Articulation; Co-speech gestures; Prosody; Speech; Speech-motor coupling.

Fig. 1

Correlation of timing between F0 maximum and co-speech gesture apex, where both the time of gesture apex and time of max f0 are relative to the target word onset.

Fig. 2

TB horizontal (x axis) and vertical (y axis) displacement during target achievement for /i, o, a/. More negative values on the y-axis correspond to lower positions of the TB and more negative values on the x-axis correspond to fronter positions of the TB. The plot ellipses illustrate 5% of the mean and one standard deviation of the mean.

Fig. 3

GAMMs of TB vertical and horizontal displacement as a function of CSG presence for /i, o, a/ across stress conditions, z-scored by subject. In (a) and (b) the top of the plot corresponds with higher tongue posture and the bottom corresponds with lower tongue posture. In (c) and (d) the top of the plot corresponds with a fronter tongue posture and the bottom corresponds with a backer posture. Shading indicates portions of trajectory that are significantly different. x-axis time is normalized by target word.

Fig. 4

Acoustic vowel duration in stressed syllables in final stress (left) and initial stress (right) tokens.

Fig. 5

GAMMs of vertical displacement and velocity of the TT (left) and TB (right) across stress conditions, z-scored by subject. Shading indicates portions of trajectory that are significantly different. x-axis time is normalized by target word. In (a) and (c) the top of the plot corresponds to higher positions of the articulator and the bottom of the plot corresponds to lower positions of the articulator. In the velocity plots (b) and (d), values above the zero line indicate upward vertical movement and values below the zero line indicate downward vertical movement. Values near zero correspond with low velocity. Where values cross the zero line, this indicates a change in the direction of the articulator.

Fig. 6

Variability in synchronization (st dev) between maximum displacement of TB and JW in stressed syllables.

Fig. 7

Positive correlation between articulatory magnitude and stability. Lower (more negative) values in vertical displacement indicate a lower posture of the tongue, i.e., increased movement magnitude. The x-axis shows TB vertical displacement. The y-axis shows the lag between TB and JW maximum displacement (a) and TB maximum displacement and CSG apex timing (b) during the stressed vowel. All values are z-scored by subject.

Fig. 8

Reference and lip sensor placement.

Fig. 9

Tongue, lip, and jaw sensor placement.

Fig. 10

Example stills of video camera positions demonstrating target production during the gesture and no gesture conditions.