Foreground-Background Segmentation Revealed during Natural Image Viewing

Abstract

One of the major challenges in visual neuroscience is represented by foreground-background segmentation. Data from nonhuman primates show that segmentation leads to two distinct, but associated processes: the enhancement of neural activity during figure processing (i.e., foreground enhancement) and the suppression of background-related activity (i.e., background suppression). To study foreground-background segmentation in ecological conditions, we introduce a novel method based on parametric modulation of low-level image properties followed by application of simple computational image-processing models. By correlating the outcome of this procedure with human fMRI activity, measured during passive viewing of 334 natural images, we produced easily interpretable "correlation images" from visual populations. Results show evidence of foreground enhancement in all tested regions, from V1 to lateral occipital complex (LOC), while background suppression occurs in V4 and LOC only. Correlation images derived from V4 and LOC revealed a preserved spatial resolution of foreground textures, indicating a richer representation of the salient part of natural images, rather than a simplistic model of object shape. Our results indicate that scene segmentation occurs during natural viewing, even when individuals are not required to perform any particular task.

Keywords: fMRI; figure-ground; human; natural scenes; visual cortex; visual perception.

Figure 1.

Comparing the standard modeling approach and the pre-filtering modeling approach. A, In the standard modeling pipeline, different models are compared. After extracting features from the stimuli, competing feature vectors can be used to predict brain activity in an encoding procedure, whereas their dissimilarities can be used in a RSA. Finally, the model that better predicts brain responses is discussed. B, In our pre-filtering modeling approach, different filtered versions of the original stimuli are compared. Various biologically plausible filtering procedures are applied to the stimuli before compute a unique feature space according to a given fixed and easily interpretable model. In our approach, a single model is employed and the step showing the highest correlation with brain activity (or representational geometry) of each filtering procedure is used to build a post hoc correlation image. While the standard modeling approach is theoretically more advantageous, as its output is a fully computable model of brain activity, it cannot be applied when reliable explicit models of perceptual processes do not exist yet, as in the case of scene segmentation. Alternative attempts to reconstruct visual stimuli from brain activity have been previously reported using multivariate techniques (Stanley et al., 1999; Thirion et al., 2006; Miyawaki et al., 2008; Nishimoto et al., 2011).

Figure 2.

Analytical pipeline. A, Foreground enhancement test: the set of segmented stimuli is tested against a null distribution of 1000 permutations. Each permutation is built by randomly shuffling the 334 behavioral foreground masks and matching the RMS contrast of the behaviorally segmented counterpart. This analysis controls for size, location, and contrast of the foreground when testing whether behavioral segmentations explain each ROI RDM better than chance. B, Background suppression test: the correlation between brain RDMs and each step of the background filtering procedure is tested against the correlation determined by the intact stimuli. While information is filtered out, correlation can increase or decrease, depending on the sensitivity for background related information in each ROI. A progressive decay indicates that a region actually processes the background, while a significant increase suggests that background is suppressed. C, Filtering steps for the contrast or spatial frequencies filtering. D, From left to right: features for each model were extracted from the stimuli; the dissimilarity (1-Pearson’s r) between each stimulus pair was computed and aggregated in four RDMs; the obtained RDMs were normalized in a 0–1 range; finally, the four RDMs were linearly combined in the fixed model, which was then correlated to the fMRI RDM obtained from each ROI.

Figure 3.

Comparison of intact and behaviorally segmented images. The graphs show the correlation between the intact (green) and segmented versions (blue: isolated foreground; red: isolated background) of the images and brain RDMs (n = 55611). Dashed bars stand for significant correlations as resulting from the permutation test (p < 0.05, Bonferroni corrected; 1000 iterations). Asterisks indicate significant differences between correlation values (p < 0.05, Bonferroni corrected). Error bars represent the SE estimated with bootstrapping. Dashed lines represent the SNR estimate for each ROI, while gray shaded regions indicate its SE.

Figure 4.

Background suppression in the human visual system. Correlation between brain activity and contrast, low- and high-pass filtering applied to the background (blue) and, as a control, to the foreground (red). Filled dots mark significant correlations (p < 0.05, Bonferroni corrected) while colored shaded areas represent the SE estimates. Dashed lines represent the SNR estimate for each ROI, while gray shaded regions indicate its SE. Arrows stand for significant differences (p < 0.05, Bonferroni corrected) between each filtering step and correlation values for the intact version (up: background suppression; down: progressive decay). Results show that for early regions (V1–V3) background-related information is relevant, since the correlation significantly decays due to filtering (p < 0.05, Bonferroni corrected); on the other hand, V4 and LOC show an opposite effect, suggesting that background is suppressed in those regions.

Figure 5.

Correlation images. To visually represent these results, we combined the different filtering procedures (contrast, low- and high-pass filtering) of the step showing the highest correlation with the representational model from each ROI.