Searching through functional space reveals distributed visual, auditory, and semantic coding in the human brain

Abstract

The extent to which brain functions are localized or distributed is a foundational question in neuroscience. In the human brain, common fMRI methods such as cluster correction, atlas parcellation, and anatomical searchlight are biased by design toward finding localized representations. Here we introduce the functional searchlight approach as an alternative to anatomical searchlight analysis, the most commonly used exploratory multivariate fMRI technique. Functional searchlight removes any anatomical bias by grouping voxels based only on functional similarity and ignoring anatomical proximity. We report evidence that visual and auditory features from deep neural networks and semantic features from a natural language processing model, as well as object representations, are more widely distributed across the brain than previously acknowledged and that functional searchlight can improve model-based similarity and decoding accuracy. This approach provides a new way to evaluate and constrain computational models with brain activity and pushes our understanding of human brain function further along the spectrum from strict modularity toward distributed representation.

Fig 1. Transformation from anatomical space to functional space.

(A) We use shared response modeling (SRM) to transform whole-brain data into a k-dimensional shared feature space. (B) Each voxel is transformed into functional space by using its loadings on the dimensions of the shared space as its coordinates in functional space. (C) Example functional space for one subject. Color indicates position on the anterior-posterior y axis of the input anatomical space, and as can be seen voxels get reorganized in functional space. Voxels that are functionally similar but anatomically disparate can be grouped together (e.g., blue-purple and red in top left). Note that this functional space is three-dimensional for visualization purposes, but the functional space used in our analyses had 200 dimensions.

Fig 2. Enhanced performance of multivariate analysis with functional searchlight.

We calculated the percent improvement of functional searchlight over anatomical searchlight for every subject from the top-performing 1% of searchlights of each type. (A) Each dot represents the percent improvement for a subject from an example layer in the AlexNet visual network (conv2) and the KellNet auditory network (fc7_W), as well as for annotation vector decoding. Error bars depict 95% confidence intervals (CIs) from bootstrapping. Raw performance levels for each searchlight type and non-parametric chance baselines can be found in Fig 3A. (B) For the visual and auditory analyses, we visualize which voxels contained model-based information by depicting the count of the number of subjects for whom that voxel contributed to one or more of the top 1% of their functional and anatomical searchlights. For the semantic analysis, we do the same but only visualize the center voxels of the top 1% of searchlights to avoid clutter. (C) We compared functional vs. anatomical searchlight in a localizer task by attempting to classify brain activity evoked by images from six categories: bodies, faces, houses, objects, landscapes, scrambled. Each dot represents percent improvement from chance of the mean top 1% searchlight accuracy. Error bar depicts 95% CI from bootstrapping. (D) We visualize the locations of all voxels that contributed to the top-performing searchlights for category decoding.

Fig 3. Raw accuracy and percent improvement for all analyses.

(A) Average performance in the top 1% of functional and anatomical searchlights for neural network similarity, annotation vector decoding, and localizer category decoding. White bar is the functional searchlight performance, black bar is the anatomical searchlight performance. Error bars represent standard error across subjects. Chance (red lines) was computed in the neural network similarity and annotation vector decoding analysis as the mean of a null distribution estimated non-parametrically by rolling data in time, and in image classification as the theoretical chance level (1/6 categories). (B) Percent improvement of functional over anatomical searchlight in the top 1% of searchlights for neural network similarity (all layers), annotation vector decoding, and localizer category decoding. To calculate percent improvement, we first subtracted the chance level from the performance of each searchlight type. Error bars represent 95% CIs.

Fig 4. Simulation of distributed versus localized representations.

We simulated fMRI data with varying degrees of localized signal. Signal localization was varied by sampling locations of signal voxels from a Gaussian Random Field (GRF) and varying its FWHM (x-axis). In particular, a GRF is simulated on the brain and the location of the signal voxels is determined by the highest values of the GRF. A GRF with high FWHM is smoother, so large values will tend to be clustered together. Therefore, for a GRF with high FWHM, the n highest values will be localized together. These highest values get more distributed as the FWHM decreases. We ran our RSA analysis for the first fully connected layer of AlexNet (see Methods). The error bars show 95% bootstrapped CIs and the dots represent individual subject improvements. As the signal transitioned from localized to distributed, the relative gain in performance of the functional searchlight increased.