Alignment of color discrimination in humans and image segmentation networks

Abstract

The experiments allowed by current machine learning models imply a revival of the debate on the causes of specific trends of human visual psychophysics. Machine learning facilitates the exploration of the effect of specific visual goals (such as image segmentation) by different neural architectures in different statistical environments in an unprecedented manner. In this way, (1) the principles behind psychophysical facts such as the non-Euclidean nature of human color discrimination and (2) the emergence of human-like behaviour in artificial systems can be explored under a new light. In this work, we show for the first time that the tolerance or invariance of image segmentation networks for natural images under changes of illuminant in the color space (a sort of insensitivity region around the white) is an ellipsoid oriented similarly to a (human) MacAdam ellipse. This striking similarity between an artificial system and human vision motivates a set of experiments checking the relevance of the statistical environment on the emergence of such insensitivity regions. Results suggest, that in this case, the statistics of the environment may be more relevant than the architecture selected to perform the image segmentation.

Keywords: Divisive Normalization; U-Nets; artificial neural networks; chromatic adaptation; color discrimination; image segmentation; image statistics; vision models.

Figure 1

Different environments (top) and associated color statistics (bottom). Left: daylight natural illumination. Center: daylight+underwater filtering and scattering. Right: artificial achromatic scenes (flat spectral reflectances and equienergetic illuminant). The corresponding 1931 CIE xy diagrams show representative color samples from the scenes (in black) and the closest neighbours to the average chromaticity (in red). It also shows local principal components (in green and blue) and the associated ellipsoid (in orange) computed from the local Principal Component Analysis (local PCA) of the nearest neighbours to the average chromaticity.

Figure 2

Illustration of human color discrimination: tolerance to saturation for different hues. (A) Shows patches of flat spectral reflectance illuminated by sources with spectral radiances selected to cover the 1931 CIE xy diagram as seen on a standard CRT display (Malo and Luque, 2002). Black dots in the CIE xy chromatic diagram of (B) show the polar distribution of the chromaticity of the considered illuminants. The illuminants are organized as a function of hue and saturation, i.e. angle with respect to the x axis, and distance with respect to the white point respectively. For each hue [each column in the colored panel of (A)] the Euclidean distance in the chromatic diagram required to induce certain perceptual departure from the white color of the same luminance is different. That is why the insensitivity region around the white [determined by the circles in (A)] is an ellipse with certain orientation [orange dots in (B)]. The diagram in (C) displays the insensitivity regions for humans measured by MacAdam (1942) at a number of color locations over the chromatic diagram.

Figure 3

Scenes with modified illumination starting from a different original image: natural (top), underwater (center), and flat-reflectance, i.e., achromatic (bottom).

Figure 4

Illustrative U-Net architecture for image segmentation. The blocks in blue represent regular convolutional layers, and the blocks in green represent bio-inspired Divisive Normalization layers. Numbers by the layers indicate the number of features and black arrows represent the skip unions.

Figure 5

Tolerance of segmentation performance to illuminant change for different environments (regular U-Nets). The results of the natural, underwater, and achromatic environments are represented in the top, middle, and bottom rows respectively. Gray level represents the segmentation performance under different illuminations with regard to the reference performance obtained for the original scenes. Darker values represent lower performance. The curves in purple, orange and green, represent variations of the performance of 3%, 5%, and 10%, respectively. These curves define tolerance regions for performance in the chromatic diagram. The RMSE values represent the distance between the average of these tolerance regions in the artificial system and the corresponding tolerance ellipse in humans.

Figure 6

Tolerance of segmentation performance to illuminant change for different environments (U-Nets with Divisive Normalization). Same results as in Figure 5, but for the architecture with Divisive Normalization.

Figure 7

Distances to MacAdam ellipses: Histograms of the RMSE errors comparing the human MacAdam ellipses and the tolerance region of the models for 300 realizations with test subsets.