Neural systems supporting linguistic structure, linguistic experience, and symbolic communication in sign language and gesture

Abstract

Sign languages used by deaf communities around the world possess the same structural and organizational properties as spoken languages: In particular, they are richly expressive and also tightly grammatically constrained. They therefore offer the opportunity to investigate the extent to which the neural organization for language is modality independent, as well as to identify ways in which modality influences this organization. The fact that sign languages share the visual-manual modality with a nonlinguistic symbolic communicative system-gesture-further allows us to investigate where the boundaries lie between language and symbolic communication more generally. In the present study, we had three goals: to investigate the neural processing of linguistic structure in American Sign Language (using verbs of motion classifier constructions, which may lie at the boundary between language and gesture); to determine whether we could dissociate the brain systems involved in deriving meaning from symbolic communication (including both language and gesture) from those specifically engaged by linguistically structured content (sign language); and to assess whether sign language experience influences the neural systems used for understanding nonlinguistic gesture. The results demonstrated that even sign language constructions that appear on the surface to be similar to gesture are processed within the left-lateralized frontal-temporal network used for spoken languages-supporting claims that these constructions are linguistically structured. Moreover, although nonsigners engage regions involved in human action perception to process communicative, symbolic gestures, signers instead engage parts of the language-processing network-demonstrating an influence of experience on the perception of nonlinguistic stimuli.

Keywords: American Sign Language; brain; deafness; fMRI; neuroplasticity.

Fig. S1.

Behavioral data, including accuracy (Left) and reaction time (Right). Error bars represent 95% confidence intervals around each mean.

Fig. S2.

Statistical maps for each stimulus type relative to the fixation cross baseline, in each subject group. Thresholded at z > 2.3, with a cluster size-corrected P < 0.05. In the coronal and sagittal views, the right side of the brain is shown on the right side of each image.

Fig. 1.

Statistical maps for each stimulus type relative to the backward-layered control stimuli, in each subject group. Statistical maps were masked with the maps shown in Fig. S2, so that all contrasts represent brain areas activated relative to fixation baseline. Thresholded at z > 2.3, with a cluster size-corrected P < 0.05. In the coronal and sagittal views, the right side of the brain is shown on the right side of each image.

Fig. 2.

(Left) Between-condition differences for each subject group, for the contrasts with backward-layered control stimuli. No areas showed greater activation for gesture than ASL in either group. Thresholded at z > 2.3, with a cluster size-corrected P < 0.05. (Right) Between-group differences for the contrast of each stimulus type relative to the backward-layered control stimuli. No areas were found that showed stronger activation in signers than nonsigners for gesture, nor in nonsigners than signers for ASL. Thresholded at z > 2.3, with a cluster size-corrected P < 0.05. Significant between-group differences in the left IFG were obtained in a planned region of interest analysis at z > 2.3, uncorrected for cluster size.

Fig. S3.

Within-group, between-condition differences, for each stimulus condition relative to fixation. Thresholded at z > 2.3, with a cluster size-corrected P < 0.05.

Fig. S4.

Between-group differences for the contrast of each stimulus type relative to the fixation cross baseline. Thresholded at z > 2.3, with a cluster size-corrected P < 0.05.

Fig. S5.

Example stimuli. Top row: Example frames from ASL and gesture movies shown to participants in the fMRI study. A and B are from movies elicited by stop-motion videos produced by author TS (2); C are from movies elicited by clip art stimuli. Bottom row: Still images that were shown to participants in the MRI study after each ASL or gesture stimulus. For A and B, these are also representative of the objects and motion paths present in the original elicitation stimuli. Although as animated videos these were quite understandable, screen shots of these videos were not very clear. Thus, these images are reconstructions of the scenes from the movies, made using the same props as in the original movies, or very similar ones.