Sensory Modality-Independent Activation of the Brain Network for Language

Abstract

The meaning of a sentence can be understood, whether presented in written or spoken form. Therefore, it is highly probable that brain processes supporting language comprehension are at least partly independent of sensory modality. To identify where and when in the brain language processing is independent of sensory modality, we directly compared neuromagnetic brain signals of 200 human subjects (102 males) either reading or listening to sentences. We used multiset canonical correlation analysis to align individual subject data in a way that boosts those aspects of the signal that are common to all, allowing us to capture word-by-word signal variations, consistent across subjects and at a fine temporal scale. Quantifying this consistency in activation across both reading and listening tasks revealed a mostly left-hemispheric cortical network. Areas showing consistent activity patterns included not only areas previously implicated in higher-level language processing, such as left prefrontal, superior and middle temporal areas, and anterior temporal lobe, but also parts of the control network as well as subcentral and more posterior temporal-parietal areas. Activity in this supramodal sentence-processing network starts in temporal areas and rapidly spreads to the other regions involved. The findings indicate not only the involvement of a large network of brain areas in supramodal language processing but also that the linguistic information contained in the unfolding sentences modulates brain activity in a word-specific manner across subjects.SIGNIFICANCE STATEMENT The brain can extract meaning from written and spoken messages alike. This requires activity of both brain circuits capable of processing sensory modality-specific aspects of the input signals as well as coordinated brain activity to extract modality-independent meaning from the input. Using traditional methods, it is difficult to disentangle modality-specific activation from modality-independent activation. In this work, we developed and applied a multivariate methodology that allows for a direct quantification of sensory modality-independent brain activity, revealing fast activation of a wide network of brain areas, both including and extending beyond the core network for language.

Keywords: MEG; Magnetoencephalography; amodal; auditory; canonical correlation analysis; visual.

Figure 1.

Analysis pipeline. A, Temporal alignment procedure. MEG signals of auditory and visual subjects differed in length due to different presentation rates. To achieve alignment between signals of auditory and visual subjects, auditory signals were epoched into overlapping segments. Each segment's first sample corresponds to the auditory word onset, but each segment's length depends on the duration of the equivalent visual stimulus. Segments were then concatenated in original order to recover signal for the full sentence length. This way, the neural response to each word is fully taken into account in further comparisons, including in the case of short words for which stimulus late processing coincided with the next word presentation. B, Starting points for the multiset canonical correlation analysis were parcel-based neural signals for all subjects, consisting of five spatial components each. 1, Signals for all sentence trials were split into five subsets, and for cross-validation one subset of sentences was left out as test data, whereas the remaining four subsets served as training data. 2, Based on the training dataset, only an unmixing matrix was found, per parcel, defining the linear combination of the five spatial components so that the correlation across sets (subjects) and time samples was maximized. The cross-covariance was computed between all subjects' spatial components and across time collapsing over sentence trials. 3, The projection was applied to the test data to compute canonical variables for the left-out sentence trials (purple outline) for all subjects. Steps 2 and 3 were repeated for all folds until each sentence subset had been left out once, and the resulting canonical variables were concatenated until the entire signal was transformed. 4, Canonical variables were epoched according to word onsets, and for each time point a subject-by-subject correlation matrix was computed across words. Correlation between cross-modal subjects (pink outline) was interpreted as quantifying supramodal activation.

Figure 2.

Specificity of the within-modality correlated activity patterns. A, Time-resolved correlation values averaged across all visual subject pairings for a parcel in the left primary visual cortex (top) and all auditory subject pairings for a parcel in the left primary auditory cortex (bottom) before (dark-gray line) and after MCCA (blue and red lines). Light-gray lines show recomputed correlation values for 1000 random permutations of word order across subjects. Notably, signals of auditory subjects highly correlate even before word onset. This is likely due to a more varied distribution of information in the auditory signal caused by the continuous nature of auditory stimulation and, as a result, differing time points at which individual words become uniquely recognizable. The MCCA is blind to the stimulus timing and will thus find canonical variables that yield maximal correlations at any time point if possible. B, Cortical map of the spatial distribution of correlations, comparing visual modality subject pairs (red) with auditory modality subject pairs (blue). Correlation strength is expressed as the Pearson correlation coefficient averaged over a time window from 150 to 200 ms after word onset and normalized by the maximum value of that window.

Figure 3.

Supramodal correlated activity patterns. Time-resolved spatial maps of supramodal correlated activity patterns (averaged over all possible cross-modal subject pairings) in the left hemisphere. Medial views of the brain surface are depicted in the first and third rows, lateral views in the second and fourth rows. Color codes are for strength of correlation. Colored parcels were most strongly correlated between cross-modal subject pairs (nonparametric permutation test, corrected for multiple comparisons).

Figure 4.

Significant supramodal correlated activity patterns as assessed by an additional permutation test. Time-resolved spatial maps of supramodal correlated activity patterns were estimated across all words (both function and content words) and masked by significance as evaluated by a more conservative shuffling procedure. We permuted sentence order 500 times before MCCA to control for artificially increased correlations due to overfitting. Shown here are correlations at several time points for the left hemisphere. Medial views of the brain surface are depicted in the first, third, and fifth rows, lateral views in the second, fourth, sixth, and seventh rows. Color codes are for strength of correlation. Colored parcels were most strongly correlated between cross-modal subject pairs.

Figure 5.

Supramodal correlated activity patterns consistent across the majority of datasets. Supramodal correlated activity patterns of word-specific activity were consistent across the majority of datasets. A, Averaged correlation time courses (mean over all possible cross-modal subject pairings) are shown for selected parcels in the IFG (green), supramarginal gyrus (red), subcentral cortex (orange), ACC (pink), anterior middle temporal gyrus (aMTG; purple), and middle superior temporal gyrus (mSTG; blue). Time courses are shown for each dataset individually (light-colored lines) as well as averaged (dark lines). Gray shaded areas mark statistically significant time points. B, Time-resolved spatial maps of cross-modal correlations in the left hemisphere. Medial views of the brain surface are depicted in the first, third, and fifth rows, lateral views in the second, fourth, and sixth rows. For those parcels that were part of the largest nominal suprathreshold cluster tested on only the exploratory dataset, the mean correlation over all six datasets is shown. Color codes are for strength of correlation. In addition, the parcels at which the majority null hypothesis according to prevalence inference could be rejected are outlined in black.

Figure 6.

Time-resolved spatial maps of cross-modal correlations for the right hemisphere. The average correlation over all six datasets is shown. Color codes are for strength of correlation. In addition, the parcels at which the majority null hypothesis according to prevalence inference could be rejected are outlined in black.

Figure 7.

Cortical map of maximum prevalence threshold γ0 per parcel. For those parcels at which the global null hypothesis could be rejected, the mean (over time) maximum threshold is plotted, for which the null hypothesis can be rejected (α = 0.05). Given the sample size of six datasets, the number of second-level permutations, and a significance level of α = 0.05, the maximal possible threshold that can be reached is 0.5633.

Figure 8.

Cortical map of prevalence threshold γ0. For those parcels at which the global null hypothesis could be rejected, the maximum threshold is plotted, for which the null hypothesis can be rejected at a level of α = 0.05.