Using transitional information in sign and gesture perception

Abstract

For sign languages, transitional movements of the hands are fully visible and may be used to predict upcoming linguistic input. We investigated whether and how deaf signers and hearing nonsigners use transitional information to detect a target item in a string of either pseudosigns or grooming gestures, as well as whether motor imagery ability was related to this skill. Transitional information between items was either intact (Normal videos), digitally altered such that the hands were selectively blurred (Blurred videos), or edited to only show the frame prior to the transition which was frozen for the entire transition period, removing all transitional information (Static videos). For both pseudosigns and gestures, signers and nonsigners had faster target detection times for Blurred than Static videos, indicating similar use of movement transition cues. For linguistic stimuli (pseudosigns), only signers made use of transitional handshape information, as evidenced by faster target detection times for Normal than Blurred videos. This result indicates that signers can use their linguistic knowledge to interpret transitional handshapes to predict the upcoming signal. Signers and nonsigners did not differ in motor imagery abilities, but only non-signers exhibited evidence of using motor imagery as a prediction strategy. Overall, these results suggest that signers use transitional movement and handshape cues to facilitate sign recognition.

Keywords: American Sign Language; Gesture; Prediction; Pseudosign; Transitions.

Fig. 1.

Still images of the experimental items for the target-detection task: Grooming Gestures (GG) and Pseudosigns (PS). Full videos are available on OSF: https://osf.io/bzfa5/?view_only=f1ebdd4738a943279c00da5bcaa2c99a.

Fig. 2.

Sample video frames from Normal (left) and Blurred (right) pseudosign stimuli. Full videos are available on OSF: https://osf.io/bzfa5/?view_only=f1ebdd4738a943279c00da5bcaa2c99a.

Fig. 3.

Response Time (RT) parameter estimates for a linear mixed effects model examining the effect of Video Condition and Stimulus Type on RTs for each group. Gest = Grooming gesture; PSign = pseudosign; Norm = Normal videos. Stars indicate a significant difference between conditions (p < 0.05).

Fig. 4.

Contrasts of marginal linear predictions illustrating the use of transitional handshape information (Blur – Normal conditions) in the first and second halves of the experiment. The linear mixed effects model examined the effect of condition half and type (Gest = grooming gesture; PSign = pseudosign) on RTs for each group. Error bars indicate Standard Error. Stars indicate a significant effect (p < 0.05).

Fig. 5.

Percent correct (mean and standard error) on the hand Test of Ability in Movement Imagery (TAMI-h). For isolated trials, participants match imagined hand configurations with pictures of hands, while for functional imagery trials participants match imagined hand configurations with manipulable objects.