The representation of colored objects in macaque color patches

Abstract

An important question about color vision is how does the brain represent the color of an object? The recent discovery of "color patches" in macaque inferotemporal (IT) cortex, the part of the brain responsible for object recognition, makes this problem experimentally tractable. Here we recorded neurons in three color patches, middle color patch CLC (central lateral color patch), and two anterior color patches ALC (anterior lateral color patch) and AMC (anterior medial color patch), while presenting images of objects systematically varied in hue. We found that all three patches contain high concentrations of hue-selective cells, and that the three patches use distinct computational strategies to represent colored objects: while all three patches multiplex hue and shape information, shape-invariant hue information is much stronger in anterior color patches ALC/AMC than CLC. Furthermore, hue and object shape specifically for primate faces/bodies are over-represented in AMC, but not in the other two patches.

Fig. 1

Recording sites, connectivity, and color stimuli. a To identify the correct color of an object, e.g., an apple (left), local hue information (right) needs to be integrated with global shape information. b Schemes for co-representing color and object shape information in visual system. Initially, before the visual system has explicitly segmented objects, color information and object shape information are largely entangled, with individual cells participating in the coding of multiple hues and object shapes. Two main strategies could be used to represent colored-objects in an organized way: (1) segregation of color and object information into parallel channels, resulting in object shape-invariant color-selective units and color-invariant shape selective units (top); (2) formation of units sharply tuned to both color and object shape (bottom). c Coronal and sagittal slices showing location of fMRI-identified face (blue) and color patches (yellow) in one monkey (M2) targeted for recording; dark-black line indicates electrode. The most anterior color patch was not observed with fMRI using the color localizer in this animal, and was located by electrical microstimulation in ALC (bottom panel, changes in BOLD signal of the identified voxels during microstimulation shown below). d Comparison between color patches identified by color localizer (top) and by microstimulation (bottom). The contrasts are overlaid on high-resolution coronal slices. Asterisk (*) indicates the stimulation site (ALC). The anterior–posterior position of each slice in mm relative to the interaural line is given in the top right corner. e 82 images of 10 categories were used (see Supplementary Fig. 2a for all the stimuli). Each image underwent a series of transformation in hues. For each pixel of the image, luminance was kept constant, while chromatic coordinates (CIE 1960) fell on a circle with the same distance to “white” (filled circle) as the original pixel. Eight hues with different angles were used (open circles, starting from 0°, going clockwise at 45° step). A grayscale image with the same luminance and the natural color image were also presented

Fig. 2

Responses of color-patch neurons to the color stimuli. Responses of all neurons in three patches to all grayscale images, colored human faces, monkey faces, and magic cubes, sorted according to hue preference of the average response across all 82 stimuli. Color-selective cells and non-selective cells are shown separately. Responses of IT cells outside color patches and cells in face patch AM are also shown. For each cell, baseline was subtracted and the response was normalized

Fig. 3

Representation of color by color-selective neurons in color patches. a Responses of all color-selective neurons averaged across stimuli within each category to images of 8 different hues, together with gray (left-most column) and natural color (right most-column), sorted in the same way as Fig. 2. b Neural representation of colors in the activities of color-selective neurons. Shown are two-dimensional plots of the results of multi-dimensional scaling (MDS) analyses conducted for neurons in three color patches. Responses to each color condition were averaged across all mammal images (humans faces, monkey faces, and mammal bodies; these were selected because color tuning was most consistent between these three categories across all three patches, see Supplementary Fig. 3b). Original color is indicated by a disk of mixed color. c Neural distances of each hue to its two neighboring hues, for all three patches, computed using population responses of color-selective cells. Error bars represent s.d. of 20,000 iterations of bootstrapping. Inhomogeneity was quantified by computing the ratio between the s.d. of the 8 bars and the mean of the 8 bars: 0.21 ± 0.02 for CLC; 0.12 ± 0.02 for ALC; 0.35 ± 0.02 for AMC (p < 0.001 between CLC and AMC; p < 0.001 between ALC and AMC; p = 0.0103 between CLC and ALC, 20000 iterations of bootstrapping, see Methods). d Population similarity matrices of 10 color conditions in three color patches. A 10×10 matrix of correlation coefficients was computed between responses of all color-selective neurons averaged across objects. e For five different types of objects: grapes, watermelon, birds, gratings, and rubik’s cube, the number of AMC cells preferring each of the eight hues was counted. In all five cases, the distribution was significantly different from homogeneity (chi-square test: p < 0.001; χ²(7) = 26.3, 28.6, 31.0, 38.6 and 28.4, respectively)

Fig. 4

Representation of object shape by color-selective neurons in color patches. a Neural representation of object shapes in the activities of color-selective neurons for three patches, shown as two-dimensional MDS plots. Responses to each object shape were averaged across 8 hues. b Decoding accuracies for identifying one object out of 82 objects based on population responses in three color patches, averaged across object identities within each category (see Methods). Error bars represent s.e. Dashed lines indicate chance level (1/82 = 1.2%). c Raster plot showing responses of an AMC neuron to colored human faces of 11 identities at three views: frontal, left-, and right-profile (Supplementary Fig. 2i). d The same response in c averaged across 8 hues, showing strong correlation between left- and right-profiles, but not between frontal and profile views. e Correlation between responses to left and right profile views was computed across identities for each cell. Mean and s.e. of all neurons in three patches are plotted (n = 26 CLC cells; n = 40 ALC cells; and n = 44 AMC cells). Student’s t-test was used to determine statistical significance between patches (* p < 0.05, ** p < 0.01). f Population similarity matrices of 11 identities ×3 views in three color patches. The paradiagonal stripes in AMC indicate high correlation between responses to mirror-symmetric views of the same identity (red arrows)

Fig. 5

Co-representation of hue and object identity in color patches. a Comparison of MDS plots of responses to all human face images in all three color patches and outside color patches. For clarity, the original natural color images are not shown. In ALC and AMC, the images were clearly grouped according to hue, while in CLC, this grouping is less clear. Outside the color patches, images were grouped according to identity, but not hue. b Population similarity matrices computed from responses to human face images in three color patches and outside color patches. Correlation coefficients were computed between responses to 11 identities and 8 hues. c Hue information and identity information for images of 10 categories in three color patches and outside color patches. Hue information was quantified as the mean correlation between responses to images with the same hue, but different identity within the same category, while identity information was quantified as the mean correlation between responses to images of the same identity with different hues. Note that here we are quantifying shape-invariant hue tuning, which will be affected by both shape tuning and color tuning; in particular, cells with strong shape tuning will show low shape-invariant hue tuning, even if they have perfectly consistent hue tuning across shapes. Error bars represent s.d. of 2000 iterations of bootstrapping. Statistical significance was determined between hue and identity information for each category in three color patches and outside color patches (* p < 0.05, **p < 0.01). d Amplitudes of hue and identity information for three patches and outside color patches, computed over a 50 ms sliding time window, were averaged across all 10 categories. Shaded regions indicate s.d. estimated by 2000 iterations of bootstrapping. e Co-representation of hue and category in all three patches and outside color patches. Responses of each cell were averaged across different identities within a category. A matrix of correlation coefficients was computed between responses to 10 categories* 8 hues. f and g Same as c and d, but for hue and category information quantified by matrices in e, **p < 0.01

Fig. 6

Decoding shape-invariant color and color-invariant shape from color patches. a SVM models were trained to classify hues independent of shape. The population response of a set of randomly-selected units was used as the input to each model. Half of the trials were used for training and the remaining half for cross-validation. Shape-invariant hue could be significantly better decoded by AMC and ALC populations than by CLC (for 50 units, p < 0.01 for both comparisons, 2000 iterations of bootstrapping). Furthermore, ALC showed better overall decoding than AMC (p = 0.013). Dashed line indicates chance level (1/8 = 12.5%). Results are averages across 2000 iterations of random sampling. Errorbars represent s.d. b similar to (a), but only quantifies decoding accuracy for two hue categories: red and yellow. Decoding based on AMC is better than ALC, but not significant (p = 0.185). c similar to (a), but for a combined population of anterior color-patch neurons. d similar to (a), but for hue-invariant shape decoding. CLC is significantly better than ALC and AMC (for 50 units, p < 0.01 between CLC and ALC, p = 0.028 between CLC and AMC). Furthermore, neurons outside color patches showed better performance than color-patch neurons, but only significantly better than ALC and AMC (For 25 units, p < 0.01 between outside and ALC, p = 0.019 between outside and AMC and p = 0.186 between outside and CLC). Dashed line indicates chance level (1/82 = 1.2%)

Fig. 7

Analysis of single-cell responses. a Responses of 12 example neurons to the full stimulus set. Each row represents one color condition, and each column represents one object shape. b-g Two-way ANOVA analysis examining main effects of shape and hue, as well as interactions. Two-way ANOVA analysis with 8 levels of hue and 82 levels of shape was performed on responses of each individual neuron. b Relationship between explained variances by two main effects for all neurons. Lines represent linear fits to cells in each patch. c Distribution of shape preference in all three patches and outside the color patches, defined by the explained variance by shape divided by the sum of both main effects. Arrows indicate population averages. d similar to (c), but using only coarse shape categories as shape variables. e similar to (c), but using only fine shapes within each shape category as shape variables. ANOVA analysis was carried out for each shape category independently, with 8 levels of hue and n levels of shape (n = number of shapes within this shape category). For each neuron, shape preference was computed and averaged across categories. f For 2-way ANOVA with 8 levels of hue and 82 levels of shape, F-values for both main effects are plotted against each other in log-scale. Gray lines indicate significance level (p = 0.001). g Distribution of F-values for the interaction between hue and shape. Gray dashed-lines indicate significance level (p = 0.001). h For each cell in ALC, we determined the best hue and the worst hue based on responses to 8 hues averaged across 82 object shapes. MDS analyses were conducted on shape responses for the best hue or the worst hue of each cell. Two MDS plots are shown at the same scale. i For each cell, the ratio between standard deviations of shape responses at the worst hue and the best hue was computed. If the cells were linearly adding hue and shape, the two standard deviations should be identical. Therefore, the ratio between these two reflects the extent of nonlinearity in the interaction of hue and shape. **p < 0.01, Student’s t-test

Fig. 8

Extending the conventional view of IT: a theory of color processing in IT. a-c Schematic summary of the co-representation of hue and object shape in three color patches. Here, each oval represents the receptive field of one “idealized” color neuron in the 2-d object space spanned by hue and object shape. d Conventional view of IT predicts that the major transformation of colored-object representation from posterior to anterior IT is the generation of invariance to accidental changes (e.g., view). Here each ellipsoid represents the receptive field of one “idealized” neuron in the 3-d object space spanned by hue, shape, and view. For both posterior and anterior IT, two dimensional slices at a fixed “view” should look the same as the schematic for CLC (a)