Object knowledge representation in the human visual cortex requires a connection with the language system

Abstract

How world knowledge is stored in the human brain is a central question in cognitive neuroscience. Object knowledge effects have been commonly observed in higher-order sensory association cortices, with the role of language being highly debated. Using object color as a test case, we investigated whether communication with the language system plays a necessary role in knowledge neural representation in the visual cortex and corresponding behaviors, combining diffusion imaging (measuring white-matter structural integrity), functional MRI (fMRI; measuring functional neural representation of knowledge), and neuropsychological assessments (measuring behavioral integrity) in a group of patients who suffered from stroke (N = 33; 18 with left-hemisphere lesions, 11 with right-hemisphere lesions, and 4 with bilateral lesions). The structural integrity loss of the white-matter connection between the anterior temporal language region and the ventral visual cortex had a significant effect on the neural representation strength of object color knowledge in the ventral visual cortex and on object color knowledge behavior across modalities. These contributions could not be explained by the potential effects of the early visual perception pathway or potential confounding brain or cognitive variables. Our experiments reveal the contribution of the vision-language connection in the ventral occipitotemporal cortex (VOTC) object knowledge neural representation and object knowledge behaviors, highlighting the significance of the language-sensory system interface.

Fig 1. Flowchart of the imaging and behavioral data analysis and the lesion distribution map of the 33 patients.

The data processing workflow was as follows: For task fMRI data: ① The neural RDM (a) was constructed by correlating the activity patterns of each pair of fruits and vegetables (estimated from the task fMRI data) within a sphere (radius = 6 mm) centered on each voxel, via Pearson’s correlation distance. The object color RDM of each subject (b) was obtained from pairwise object color similarity ratings. Partial Spearman’s rank correlation was then computed between the neural and object color RDMs, controlling for low-level visual control, shape, and general semantic RDMs, to construct the object color RSA maps of healthy controls (c) and patients (g). ② In the functionally defined VOTC-color-perception mask (d), the top 300 selected voxels (i.e., voxels with the highest Fisher-Z transformed r values) in individual participants’ object color RSA maps were binarized to construct the individual-level mask (e). ③ The individual masks were overlaid across those of all healthy controls and a group-level threshold of 0.25 was applied to obtain the group-level functional mask (i.e., VOTC-color-knowledge mask, f). ④ The mean Fisher-Z transformed r value of each patient’s object color RSA map (g) within the VOTC-color-knowledge mask (f) was calculated to quantify the neural representation of object color knowledge in the VOTC. For HARDI data: ⑤ Probabilistic tractography was implemented using the estimated HARDI data (h) between each pair of ROIs (e.g., VOTC-LdlATL, i) in the native space. The resulting tracking map was normalized and standardized using the maximum voxel intensity of each image and then binarized at 0.1 to construct the individual-level mask (j). ⑥ A group-level threshold of 0.5 across all healthy controls within the explicit WM mask was applied to obtain the group-level WM mask (k). ⑦ The mean value of each patient’s FA map (l) within the group-level WM mask (k) was calculated to quantify the WM integrity of this fiber bundle. Correlation analysis: ⑧ The correlations between the WM integrity (mean FA value) of each WM tract and the neural representation strength of object color knowledge in the VOTC-color-knowledge mask across patients were calculated to identify the tracts relevant to the VOTC knowledge representation. ⑨ Object color behaviors were measured as the composite score of the grayscale picture-color word matching task (verbal) and the object color true/false judgment task (non-verbal). ⑩ ⑪ The correlations were computed between the WM integrity, the neural representation of object color knowledge, and the object color composite score, respectively, to identify the neural correlates associated with object color behavior. The upper-right panel shows the lesion distribution map of the 33 patients. The n value of each voxel denotes the number of patients with a lesion. Brain imaging results were visualized using BrainNet Viewer (version 1.7; https://www.nitrc.org/projects/bnv/; RRID: SCR_009446), MRIcron (version 1.0.20190902; https://www.nitrc.org/projects/mricron; RRID: SCR_002403), or MRIcroGL (version 1.2.20210317; https://www.nitrc.org/projects/mricrogl). Abbreviations: RDM, representational dissimilarity matrix; VOTC, ventral occipitotemporal cortex; RSA, representation similarity analysis; ROI, region of interest; HARDI, high angular resolution diffusion imaging; L, left; dlATL, dorsolateral anterior temporal lobe; WM, white-matter; FA, fractional anisotropy.

Fig 2. Analysis series 1: Relationship between VOTC-language white-matter connection integrity and VOTC object color knowledge neural representation.

(A) Design of the object color judgment task fMRI experiment. Participants viewed the grayscale pictures of fruits and vegetables (1 s) and pressed the “yes” button when the skin of the items was red, and the “no” button when it was not. (B) Defining the VOTC-color-knowledge mask in healthy controls. The left panel displays the VOTC-color-perception mask, functionally defined by contrasting chromatic stimuli to grayscale stimuli in a color perceptual localizer in 14 healthy subjects [22]. The middle panel shows the unique object color probability map, constructed by the overlap of individual ROIs (i.e., the top 300 selective voxels with the highest Fisher-Z transformed r values of the color RSA results within the VOTC-color-perception mask) across 33 healthy controls. The color bar indicates the proportion of healthy controls that had the individual ROI at each voxel. Voxels with a probability greater than 0.25 (outlined by the black lines) were retained as the VOTC-color-knowledge mask, as depicted in the right panel. (C) The left panel illustrates the reconstructed VOTC-LdlATL tract (red) and the seed regions being connected (VOTC, orange; LdlATL, green). The right panel shows the raw scatter plot between the mean FA values of the VOTC-LdlATL tract and the VOTC neural representation (Fisher-Z transformed r values) and the partial Spearman’s rho value (controlling for TLV). Note that negative values for the RSA results might be difficult to interpret; when these values were set to zero, the effect remained significant (partial rho = 0.54, p < 0.01). (D) Validation of the VOTC-LdlATL tract after controlling for GM damage at both ends and TLV. (E) Validation of the VOTC-LdlATL tract after controlling for the VOTC-occipital pole tract integrity and TLV. The reconstructed VOTC-occipital pole tract (red) and the seed regions (VOTC, orange; occipital pole, yellow) being connected are shown in the left panel. The data underlying this figure are available in S1 Data. Brain imaging results were visualized using BrainNet Viewer (version 1.7; https://www.nitrc.org/projects/bnv/; RRID: SCR_009446), or MRIcroGL (version 1.2.20210317; https://www.nitrc.org/projects/mricrogl). Abbreviations: VOTC, ventral occipitotemporal cortex; L, left; dlATL, dorsolateral anterior temporal lobe; ROI, region of interest; TLV, total lesion volume; GM, gray-matter; FA, fractional anisotropy.

Fig 3. Analysis series 2: Relationship between VOTC-language white-matter connection integrity and object color knowledge behavior in patients.

(A) Design and raw accuracy in the out-scanner neuropsychological tests, including object color behavioral tasks (grayscale picture-color word matching (verbal), object color true/false judgment (non-verbal)) and control tasks (word-picture matching, color patch matching). Dot plots show raw accuracies of each participant (bars denote mean values ± 1 standard error). (B, C) The relationships between two neural measures (the neural representation of object color knowledge in the VOTC-color-knowledge mask, and the VOTC-LdlATL tract integrity) and object color behavior (the composite score across the verbal and non-verbal object color tasks). The scatter plots are shown for visualization purposes, as we removed an outlier patient (ID 006, composite score = −10.45, 4.3 SD below the patient average; mean FA value = 0.53) from the plots, the presence of which greatly increased the range of the x-axis and distorted the scatter plot. Importantly, the partial rho and p-values shown in the figures were computed on the basis of data from all the patients, after controlling for TLV. The correlations after excluding this outlier patient were similar for the neural representation (partial rho = 0.19, p = 0.30) and for the VOTC-LdlATL tract (partial rho = 0.48, p = 0.007). (D) Validation analyses of the VOTC-LdlATL tract, controlling for confounding variables and TLV. The data underlying this figure are available in S1 Data. Brain imaging results were visualized using BrainNet Viewer (version 1.7; https://www.nitrc.org/projects/bnv/; RRID: SCR_009446), or MRIcroGL (version 1.2.20210317; https://www.nitrc.org/projects/mricrogl). Abbreviations: VOTC, ventral occipitotemporal cortex; L, left; dlATL, dorsolateral anterior temporal lobe; TLV, total lesion volume; FA, fractional anisotropy; GM, gray-matter.

Fig 4. In-depth analyses of the VOTC-LdlATL connection.

(A) Effects of subsections along the VOTC-LdlATL tract. The tract was divided into the three equal subsections along the y-axis (top panel): posterior, middle, and anterior. The bottom panels show the scatter plots of the FA values of these subsections with the VOTC object color neural representation and the composite score of object color behaviors, respectively. (B) Specificity of the dorsolateral subregion of the ATL. The ATL subregions (dorsal, lateral, and ventral-medial) were obtained from Hung and colleagues [43], who parcellated the ATL on the basis of coactivation clustering; the two dorsal subregions in that study were combined here into a single dorsal subregion. The top panel displays the seed regions for probabilistic tractography and the corresponding reconstructed white-matter tracts. The bottom panels show the scatter plots of FA values of the reconstructed white-matter tracts with the VOTC object color neural representation and the composite score of object color behaviors, respectively. The partial rho and p-values shown in the figures were computed with all the patients, after controlling for total lesion volume. Similar to Fig 3, the behavioral outlier patient was removed from the scatter plots with object color behavior. The data underlying this figure are available in S1 Data. Brain imaging results were visualized using MRIcroGL (version 1.2.20210317; https://www.nitrc.org/projects/mricrogl). Abbreviations: VOTC, ventral occipitotemporal cortex; L, left; dlATL, dorsolateral anterior temporal lobe; FA, fractional anisotropy.