Unification of sentence processing via ear and eye: an fMRI study

Abstract

We present new evidence based on fMRI for the existence and neural architecture of an abstract supramodal language system that can integrate linguistic inputs arising from different modalities such that speech and print each activate a common code. Working with sentence material, our aim was to find out where the putative supramodal system is located and how it responds to comprehension challenges. To probe these questions we examined BOLD activity in experienced readers while they performed a semantic categorization task with matched written or spoken sentences that were either well-formed or contained anomalies of syntactic form or pragmatic content. On whole-brain scans, both anomalies increased net activity over non-anomalous baseline sentences, chiefly at left frontal and temporal regions of heteromodal cortex. The anomaly-sensitive sites correspond approximately to those that previous studies (Michael et al., 2001; Constable et al., 2004) have found to be sensitive to other differences in sentence complexity (object relative minus subject relative). Regions of interest (ROIs) were defined by peak response to anomaly averaging over modality conditions. Each anomaly-sensitive ROI showed the same pattern of response across sentence types in each modality. Voxel-by-voxel exploration over the whole brain based on a cosine similarity measure of common function confirmed the specificity of supramodal zones.

Figure 1

The synthetic hemodynamic response function and its relationship to words in stimulus sentence. For both print and speech stimuli, the synthetic HRF peaks 5 seconds after the onset of the verb. Annotation reflects the two word per second presentation rate for print stimuli.

Figure 2

Intersection and differences in activation for evoked responses to non-anomalous sentences in each modality (p < .00001, uncorrected). Areas coded in green are activated above threshold in speech only, areas in red are activated in print only, and yellow indicates areas that are active in both modalities (conjoint probability < 1.0e-10). Montreal Neurologic Institute (MNI) z-coordinate is indicated on each slice.

Figure 3

Map of standardized activations for the contrast between anomalous with non-anomalous sentences, collapsing across anomaly types and averaging over modality (p < .05, FDR corrected). Hot colors indicate regions more active for anomalous sentences; cold colors show areas more active for non-anomalous sentences. MNI z-coordinate is indicated on each slice.

Figure 4

Conjunction map indicating areas more responsive (p < .05, uncorrected) to (A) syntactically anomalous sentences than to non-anomalous sentences or to (B) pragmatically anomalous sentences than to non-anomalous sentences, in speech (green), or print (red), or both (yellow; conjoint probability < .0025). MNI z-coordinate is indicated on each slice.

Figure 4

Conjunction map indicating areas more responsive (p < .05, uncorrected) to (A) syntactically anomalous sentences than to non-anomalous sentences or to (B) pragmatically anomalous sentences than to non-anomalous sentences, in speech (green), or print (red), or both (yellow; conjoint probability < .0025). MNI z-coordinate is indicated on each slice.

Figure 5

Mean standardized activation scores (±sem) for each of 6 sentence types in 6 empirically derived regions of interest (6mm radius spheres, encompassing 123 voxels each). Note that y-axes are all on the same scale, although differently centered. ROIs are centered on the focal activation within anatomic regions known to respond to linguistically challenging material. ROI centers in MNI coordinates are: dIFG (−46, 16, 24); vIFG (−44, 26, 12); STG (−54, −36, 8); IPL (−58, −38, 24); MTG (−66, −10, −22); OT (−32, −40, −16).

Figure 6

Results of applying the cosine similarity metric to the whole brain. Colored voxels indicate high similarity as determined by high cosine values (≥ .6) for the angle between the speech vector, specified by values of evoked responses to three sentence conditions (Pragmatic Anomaly, Syntactic Anomaly, and No Anomaly), and the corresponding print vector.