Emergence of Visual Center-Periphery Spatial Organization in Deep Convolutional Neural Networks

Abstract

Research at the intersection of computer vision and neuroscience has revealed hierarchical correspondence between layers of deep convolutional neural networks (DCNNs) and cascade of regions along human ventral visual cortex. Recently, studies have uncovered emergence of human interpretable concepts within DCNNs layers trained to identify visual objects and scenes. Here, we asked whether an artificial neural network (with convolutional structure) trained for visual categorization would demonstrate spatial correspondences with human brain regions showing central/peripheral biases. Using representational similarity analysis, we compared activations of convolutional layers of a DCNN trained for object and scene categorization with neural representations in human brain visual regions. Results reveal a brain-like topographical organization in the layers of the DCNN, such that activations of layer-units with central-bias were associated with brain regions with foveal tendencies (e.g. fusiform gyrus), and activations of layer-units with selectivity for image backgrounds were associated with cortical regions showing peripheral preference (e.g. parahippocampal cortex). The emergence of a categorical topographical correspondence between DCNNs and brain regions suggests these models are a good approximation of the perceptual representation generated by biological neural networks.

Figure 1

Hierarchical correspondences between layers of DCNN and brain regions of interest along ventral visual pathway. (A) For each image, the activation of units in each of the 5 convolutional layers are vectorized. RDM representation for each layer is created by computing the pairwise distance of these image specific vector patterns (1-Pearson Corr). Then fMRI RDM representations in EVC, Fusiform, IT and PHC areas are compared with the RDM representations of each convolutional layer of Hybrid-CNN by computing Spearman’s correlations. (B) Neural representations along ventral visual pathway. RDM matrices, and 2D multidimensional scaling visualization of stimuli depicted for early visual cortex (EVC), fusiform gyrus (Fusiform), inferior temporal cortex (IT) and parahippocampal cortex (PHC). (C) The correlation values for brain ROIs and layers of DCNN are depicted with bar plots. The error bars indicate the standard error of the mean and the stars above each bar indicates significant correlation above zero (N = 15, P < 0.05, Bonferroni-corrected). The noise ceiling for each brain area is reported on the right side of the panel. The pictures used in this figure are not examples of the stimulus set due to copyright.

Figure 2

Creating topographical correlation maps. We extract the 3D activation patterns from the network convolutional layers. The first 2 Dimensions have a spatial relation with the image space (width and height). At each (x, y) position in feature maps, we extract a pattern vector with the length equivalent to the depth and construct the RDM matrix from the neural network activity patterns at each (x, y) location. Comparison of these RDM matrices with a brain ROI RDM results in a 2D correlation map which we then up-sample it to the image size (topographical map). The pictures used in this figure are not examples of the stimulus set due to copyright.

Figure 3

Topographical correspondence between convolutional layers of DCNNs and human ventral visual regions. For each brain-model mapping (EVC, Fusiform, IT, PHC), the first five maps show the correlational topographical maps between each convolutional layer and the brain ROI; the second five maps show the corresponding significance maps (two-sided sign permutation tests, cluster defining threshold P < 0.01, and corrected significance level P < 0.05). The topographical correlation maps in this figure are computed following the method depicted in Fig. 2. For detailed description of RDM computations and correlations please see the Method section.