Cortical Encoding of Manual Articulatory and Linguistic Features in American Sign Language

Abstract

The fluent production of a signed language requires exquisite coordination of sensory, motor, and cognitive processes. Similar to speech production, language produced with the hands by fluent signers appears effortless but reflects the precise coordination of both large-scale and local cortical networks. The organization and representational structure of sensorimotor features underlying sign language phonology in these networks remains unknown. Here, we present a unique case study of high-density electrocorticography (ECoG) recordings from the cortical surface of profoundly deaf signer during awake craniotomy. While neural activity was recorded from sensorimotor cortex, the participant produced a large variety of movements in linguistic and transitional movement contexts. We found that at both single electrode and neural population levels, high-gamma activity reflected tuning for particular hand, arm, and face movements, which were organized along dimensions that are relevant for phonology in sign language. Decoding of manual articulatory features revealed a clear functional organization and population dynamics for these highly practiced movements. Furthermore, neural activity clearly differentiated linguistic and transitional movements, demonstrating encoding of language-relevant articulatory features. These results provide a novel and unique view of the fine-scale dynamics of complex and meaningful sensorimotor actions.

Keywords: deaf; direct electrical stimulation; electrocorticography; sensorimotor control; sign language.

Figure 1:. Individual electrodes show sensorimotor selectivity for relevant linguistic features of ASL.

(A) Linear regression R² values showing electrodes that respond significantly to location and handshape features. The yellow line marks the central sulcus. Filled electrodes passed a statistical cutoff for the regression analysis (p<0.01, Bonferroni corrected for multiple comparisons across electrodes). (B) Comparison of single electrode activity for movements occurring at hand and face locations. (C-E) Single electrodes respond selectively to hand (C,D) and face (E) locations, controlling for handshape. Some responses begin at movement onset (video frame when the participant begins to move; C), and others begin up to ~1sec before movement onset (D,E). (F) Comparison of single electrode activity for ‘O’ and ‘S’ handshapes, which are physically similar. (G-I) Single electrodes respond selectively to ‘O’ (G,H) and ‘S’ (I) handshapes, controlling for location. See Figure S1, Table S1, and Video S1.

Figure 2:. Hierarchical clustering of sensorimotor features.

Unsupervised clustering of neural data reveals hierarchical structure of linguistically-defined phonological features. (A) Fingerspelling location is distinct from all other locations. Larger branches are generally organized according to location, while the smallest distances reflect handshape similarity. (B-C) Neural data was used to classify four locations. The timecourse of classification accuracy revealed a peak prior to movement onset (B). At the peak, all four locations are classified above chance, with poorer performance for “neutral” (C). (D-E) Neural data was used to classify eight handshapes. The timecourse of classification accuracy revealed a peak just after movement onset (D). At the peak, handshape confusions were largely driven by sensorimotor similarity (E). Light grey in B and D reflects chance distributions. See Figure S2 and Table S1.

Figure 3:. Direct electrical stimulation reveals sensorimotor and language organization.

Sites with evoked motor, sensory, and language errors are marked on the participant’s brain. Sites with multiple effects have markers offset for visualization. Black line indicates the central sulcus. No sites showed only motor hand effects.

Figure 4:. Localization of linguistic features of sign.

Transitional movements (not part of lexical signs or fingerspelling) were compared to linguistic movements. (A-C) Example electrodes showing timecourses for linguistic signs and transitional movements (matched for handshape). Electrodes are labeled in D. (D) The peak difference between transitional and linguistic movements reveals generally larger responses to linguistic signs throughout pre-central, post-central, and parietal cortex. The dorsal portion of post-central gyrus and anterior SMG respond more strongly to transitional movements. (E) The peak latency of the difference between transitional and linguistic movements reveals an anterior-to-posterior gradient of activity across sensorimotor cortex. (F) Producing real signs evokes stronger activity in most areas except posterior dorsal parietal cortex. Pseudo signs are matched to real signs for handshape. See Figure S3.

Figure 5:. Lexical frequency and age of acquisition (AoA) effects on stimulus- and response- locked activity.

(A) At ~1000ms after stimulus onset, there is a negative effect of frequency and a positive effect of AoA, explaining a maximum of ~8% of the variance on average. Box plots beneath axes indicate interquartile range and 1.5 times interquartile range for response onset (red), stimulus offset (green), and movement onset (blue). (B) Response-locked (production) neural activity was also modulated by frequency and AoA, with a broader timecourse peaking around the onset of each movement. (C) Electrodes significantly modulated by frequency and AoA during the post-stimulus window were primarily localized to dorsal prefrontal cortex. A subset of the same electrodes was modulated during production. See Figure S4 and Figure S5.