Sensory-motor transformations for speech occur bilaterally

Abstract

Historically, the study of speech processing has emphasized a strong link between auditory perceptual input and motor production output. A kind of 'parity' is essential, as both perception- and production-based representations must form a unified interface to facilitate access to higher-order language processes such as syntax and semantics, believed to be computed in the dominant, typically left hemisphere. Although various theories have been proposed to unite perception and production, the underlying neural mechanisms are unclear. Early models of speech and language processing proposed that perceptual processing occurred in the left posterior superior temporal gyrus (Wernicke's area) and motor production processes occurred in the left inferior frontal gyrus (Broca's area). Sensory activity was proposed to link to production activity through connecting fibre tracts, forming the left lateralized speech sensory-motor system. Although recent evidence indicates that speech perception occurs bilaterally, prevailing models maintain that the speech sensory-motor system is left lateralized and facilitates the transformation from sensory-based auditory representations to motor-based production representations. However, evidence for the lateralized computation of sensory-motor speech transformations is indirect and primarily comes from stroke patients that have speech repetition deficits (conduction aphasia) and studies using covert speech and haemodynamic functional imaging. Whether the speech sensory-motor system is lateralized, like higher-order language processes, or bilateral, like speech perception, is controversial. Here we use direct neural recordings in subjects performing sensory-motor tasks involving overt speech production to show that sensory-motor transformations occur bilaterally. We demonstrate that electrodes over bilateral inferior frontal, inferior parietal, superior temporal, premotor and somatosensory cortices exhibit robust sensory-motor neural responses during both perception and production in an overt word-repetition task. Using a non-word transformation task, we show that bilateral sensory-motor responses can perform transformations between speech-perception- and speech-production-based representations. These results establish a bilateral sublexical speech sensory-motor system.

Figure 1. Behavioral tasks and example neural activations

a) Subjects were auditorily presented with a CVC single syllable word and instructed to perform one of three tasks on interleaved trials: i. Listen-Speak - Listen to the word, visual prompt ‘Listen’. After a 2 s delay repeat the word, visual prompt ‘Speak’. ii. Listen-Mime – Listen as for ii. After delay, mime speaking the word, visual prompt ‘Mime’. iii. Listen - Passively listen to the word, visual prompt ‘:=:’. Auditory and motor timelines (below). b) Example time-frequency spectrograms of ECoG activity normalized at each frequency to the Baseline power during visual prompt. Sensory-motor (S-M): Significant activity during the auditory and movement epochs in Listen-Speak and Listen-Mime tasks. Production (PROD): Significant activity during both movement epochs. Auditory (AUD): Significant activity during each task epoch with auditory stimuli.

Figure 2. Topography of neural responses and bilateral activation

a) Significant task-related activations within individual subject brains for left (S3,S4,S7,S8,S11,S14), right (S1,S2,S5,S6,S9,S12,S15), or both (S10,S13,S16) hemispheres. Bilateral coverage (light blue box). Electrodes with significant high gamma activity (70–90 Hz): AUD (green), PROD (blue), and S-M (red). AUD and S-M activations (red with green) were often present on the same electrode. Electrodes without significant activation (grey). Triangles denote example activations from Fig 1b, and squares (S16) denote example spectrograms in Fig 2c. b). Significant electrodes projected onto population average left and right hemispheres, color convention as a). Electrode sizes have been increased for illustrative purposes (actual sizes - Fig S4. c) Neural spectrograms for example S-M electrodes in left (upper) and right (lower) hemispheres of S16 during Listen-Speak, Listen-Mime, and Listen tasks. d). Population average neural response profiles for each class of electrodes. Shaded regions indicate SEM values across electrodes. Go Cue and average production response onset (grey arrows).

Figure 3. Neural decoding of words

a) Confusion matrices for a 7-way linear classifier using neural responses. AUD electrodes (top panel). PROD electrodes (middle panel). S-M electrodes (bottom panel). Performance is thresholded at chance performance, p = 0.14, for display purposes only. b) Classification performance for increasing numbers of electrodes. Chance performance (dotted). c) Classification performance for S-M electrodes in the left (dark red) and right (light red) hemispheres. Chance performance (dotted). Online Methods presents S-M results by response epoch.

Figure 4. Listen-Speak Transformation task

a) In the Listen-Speak Transformation task, subjects have to transform a non-word they hear into a non-word they speak according to a simple rule. Subjects were first presented with a visual cue: ‘match listen’ or ‘mismatch listen’ that instructed the rule that determined the non-word to say in response to the word they heard. On ‘match’, trials the rule was to repeat the non-word they heard. On ‘mismatch’ trials, they should say the non-word they did not hear. The non-words were ‘kig’ and ‘pob’. Subjects then heard one of the two non-words, waited for a short delay, then said the appropriate non-word in response to the ‘Speak’ cue. There were four task conditions. Kig→kig: hear ‘kig’ and say ‘kig’. Pob→pob: hear ‘pob’ and say ‘pob’. Kig→pob: hear ‘kig’ and say ‘pob’; and Pob→kig: hear ‘pob’ and say ‘kig’. b) Confusion matrices predicted by the Sensory, Motor and Transformation models. c) Confusion matrices during the Listen-Speak Transformation task. d) Model fit quantified using a Kullback-Leibler (K-L) divergence.