Brain feature maps reveal progressive animal-feature representations in the ventral stream

Abstract

What are the fundamental principles that inform representation in the primate visual brain? While objects have become an intuitive framework for studying neurons in many parts of cortex, it is possible that neurons follow a more expressive organizational principle, such as encoding generic features present across textures, places, and objects. In this study, we used multielectrode arrays to record from neurons in the early (V1/V2), middle (V4), and later [posterior inferotemporal (PIT) cortex] areas across the visual hierarchy, estimating each neuron's local operation across natural scene via "heatmaps." We found that, while populations of neurons with foveal receptive fields across V1/V2, V4, and PIT responded over the full scene, they focused on salient subregions within object outlines. Notably, neurons preferentially encoded animal features rather than general objects, with this trend strengthening along the visual hierarchy. These results show that the monkey ventral stream is partially organized to encode local animal features over objects, even as early as primary visual cortex.

Fig. 1.. Heatmaps.

(A) Example of a stimulus: a 16° × 16° test natural scene (the working image region) embedded within a larger 30° × 30° background of brown noise to prevent edge artifacts and motion cues. (B) Grid of 9° × 9° = 81 stimulus positions relative to the population RF, with the center at (0, 0)°. (C) Fixation task: Monkeys fixated on a central red dot while stimuli were presented for 100 ms at different grid positions relative to the population RF (pink dashed circle). (D to G) The simplest case of a heatmap evoked by a working image region. (D) The working image region contains a 2°-wide cartoon face placed within a brown noise background [only the working image region is shown, which remains 16° × 16° as in (A)]. This particular stimulus serves to locate offsets between RF and the grid center. (E) Spike rate response elicited by every one of the 81 positions within the working image region: This is the raw heatmap. (F) Heatmap resized and interpolated (via imresize.m, MATLAB) to the size of the working image region (16° × 16°) and (G) superimposed on the stimulus. (H) Schematic showing the approximate location of implanted chronic arrays for the V1/V2, V4, and PIT regions. (I) Examples of natural scenes used as stimuli (one image pixelated due to copyright reasons). (J) Averaged heatmaps of neuronal populations in V1/V2 (column 1), V4 (column 2), and PIT (column 3) in monkey A and monkey B generated from the natural scenes shown in (I). The heatmaps may show a gap between the edge of the image: This reflects correction for the center position of the population RF, as these differed as a function of visual area. Two monkeys with implants in V1/V2, V4, and PIT are featured in this figure, although major results were replicated across all monkeys.

Fig. 2.. Neurons across the hierarchy showed increasing focus on animal features.

(A) Example of the animal masks derived from the scene of a chimpanzee in a jungle (top) and the background mask of the same image (n = 2 animals for illustration). (B) Population heatmap activity inside (solid lines) and outside of the mask (dashed lines, ±SEM), for V1/V2 (red), V4 (orange), and PIT (blue), plotted as a function of image onset (n = 2 animals for illustration). (C) Aggregate difference between the curves, measured as the area under the curve (AUC) for the difference curve defined by the inside versus outside response curves $\text{AUC}\left[D_r(x_k, t)\right]$, for V1/V2, V4, and PIT (red, orange, and blue, n = 283, comprising responses of neuronal populations for six animals, up to 36 pictures); the observed values in dark colors are compared to the shuffled-map AUC values (lighter colors). Top plot: Animal mask, Middle plot: Food mask. Bottom plot: Plant mask. (D) Neuronal response $\text{AUC}\left[D_r(x_k, t)\right]$ per visual area, in six monkeys, for animal (black) and nonanimal (purple) representative categories. Insets show examples of the content of the animal and nonanimal masks.

Fig. 3.. Feature to object.

(A) Animal mask of a natural scene of a mother monkey and a baby monkey. (B) Illustration of neuronal activity if it was distributed evenly within the object outline, resulting in FTO of 1. (C) PIT and V1/V2 population heatmap and shuffled heatmap in monkey A. (D) FTO(animal) of PIT and V1/V2 for heatmaps in (C). (E) Top: FTO ratio of monkey A in V1/V2 (red), V4 (orange), and PIT (blue) for animal, eye, head, and other natural object; the observed FTO values in dark colors. Bottom: In comparison, the FTO values in shuffled map (lighter colors) for the example. Monkeys A and B used for this analysis. Results replicated in monkeys C and D.

Fig. 4.. Population and individual-site heatmaps.

(A) Example stimulus image (left). Population-level heatmaps for V1/V2 (top left) and PIT (bottom left) shown alongside representative individual site-level heatmaps for V1/V2 (top row) and PIT (bottom row). Each site’s heatmap shows its correlation to the population map (monkey A). (B) Swarm plots showing the distribution of correlations between individual site-level maps and their respective population maps for V1/V2, V4, and PIT, across all animals. Central value shows the mean correlation. (C) Fraction of animal-feature–preferring sites across V1/V2, V4, and PIT. Selectivity was defined by a significant semantic selectivity index (SSI) favoring animal-related masks (P < 0.01, observed fractions in black, gray shows fractions after FDR correction of P values). (D) SSI for animal-related and nonanimal-related masks across individual neuronal sites in V1/V2 (red), V4 (orange), and PIT (blue). Lines connect paired animal and nonanimal SSIs for a given site. All six monkeys were used for these analyses.

Fig. 5.. Evaluation of deep neural networks on brain similarity.

(A) Example of feature maps (“heatmaps”) from artificial neural networks (ANNs) sampled in the early, middle, and late layers. (B) Aggregate difference between the curves as a function of depth. (C) Trend as a function of semantic mask type, per different ANNs (color). (D) Pearson correlation coefficients measuring the similarity of each ANN to neural data. Each dot represents an individual model, ranked along the y-axis by its correlation with monkey trends. Higher values indicate stronger alignment with the monkey neural hierarchy. Dots are color-coded to match the top and bottom-ranked model names, for visual clarity. (E) Scatterplot comparing the slopes of feature selectivity trends for animal-related features (x axis) versus nonanimal-related features (y axis) across monkeys and ANNs. Each point represents a single model or the monkey data, with the x axis capturing how feature selectivity for animals increases across layers and the y axis capturing the trend for nonanimal features. Models closer to the monkey point exhibit trends that better align with monkey neural data. Error bars indicate the variability of slopes for each feature type. The gray line shows the least-squares regression line to the scatter plot. (F) Slope of FTO versus slope of FTO in control for animal masks (dark colors), and other natural objects (lighter colors) in monkeys, CORnet-S and ViT-base. All of the monkeys were used in these analyses.

Fig. 6.. Free viewing and automated saliency maps.

(A) Example scene. (B) Example of eye positions tracked over the scene to create fixation maps. (C) Examples of viewing maps on the same scene (i), along with neuronal heatmaps from V1/V2, V4, and PIT (ii), and algorithmic saliency maps [GBVS; Itti-Koch saliency; and fast, accurate, and size-aware salient object detection (FASA implemented in DeepGaze) (iii); monkeys A and B used]. (D) Pearson correlation values relating all neuronal heatmaps (V1/V2, V4, and PIT), all fixation maps (eye), and all saliency maps (gbvs, itti, and dg). The diagonal cells show the monkey-monkey correlations in neuronal heatmaps and the monkey-monkey correlations in fixation maps. (E) Similarity of neuronal heatmaps across monkeys for V1/V2 maps, for V4 maps, and for PIT maps [observed (Obs) values: dark red, orange, and blue; shuffled (Shu) neuronal heatmaps: light red, orange, and blue]; each point shows one scene. (F) Similarity of neuronal heatmaps to fixation maps, per area. (G) Similarity between fixation maps to neuronal heatmaps and to saliency algorithms. Monkeys A and B were used for these analyses.