The neural basis of image segmentation in the primate brain

Abstract

Image segmentation is a fundamental aspect of vision and a critical part of scene understanding. Our visual system rapidly and effortlessly segments scenes into component objects but the underlying neural basis is unknown. We studied single neurons in area V4 while monkeys discriminated partially occluded shapes. We found that many neurons tuned to boundary curvature maintained their shape selectivity over a large range of occlusion levels as compared to neurons that are not tuned to boundary curvature. This lends support to the hypothesis that segmentation in the face of occlusion may be solved by contour grouping.

Keywords: monkey; object recognition; shape representation; ventral pathway.

Figure 1

Responses of a V4 neuron tuned to boundary curvature. Shape preference was characterized using a set of 43 shapes (columns) presented at 8 rotations (rows) in a passive fixation task. Some shapes (1, 36 and 43) were shown at fewer rotations due to rotational symmetry. The background intensity of each icon depicts the average response to that shape. Responses were strongest for shapes containing a sharp convexity to the lower right. Shapes highlighted by red (preferred) and blue (non-preferred) squares were chosen as the discrimination stimuli for the behavioral task (see Figure 4). Previously published in Kosai et al, (2014).

Figure 2

Suppression of preferred responses under partial occlusion context. A. Angles θ and φ are real contours for the crescent in isolation (left); when the crescent is adjoined by a contextual stimulus (right), these angles are interpreted as accidental contour features formed at the T-junctions between the occluding (blue) and occluded (red) shapes and are perceptually less salient. B–D. Example V4 neuron that exhibits suppressed encoding of accidental contours. Average responses of an example neuron to: four primary shapes (B), context stimuli (C) and combination stimuli (D) presented at 8 orientations (columns) are shown in grayscale. Blue bars in the lower right corner of each icon indicate standard errors of the mean (SEM). B. Primary shapes with a sharp convexity at the bottom of the shape (225°–315°) evoked strong responses from this cell. C. Contextual stimuli presented in the non-preferred color evoked weak responses. D. Preferred primary shape responses (B: 225°–315°) were strongly suppressed in the presence of corresponding contextual stimuli. E. Schematic of how a visual scene is encoded in area V4. The left panel shows an example visual scene with partially occluded objects. All boundaries in the image are shown in the middle panel. Real contours are shown in green. Accidental sharp convexities at T-junctions (labeled s) and accidental concavities between the T-junctions (labeled c) are shown in red. In area V4 only the real contours (green) are strongly encoded. This may serve as the first step of segmentation in the primate brain. Adapted from Bushnell et al., (2011).

Figure 3

Behavioral task. A. Sequence and duration of trial events. After acquiring fixation, monkeys viewed two shapes: the reference stimulus at the center of gaze followed by the test stimulus in the neuron’s RF. The fixation point was then extinguished and two peripheral choice targets appeared. Animals reported whether the reference and test stimuli were the same or different (match/nonmatch) with rightward and leftward saccades, respectively. New discrimination stimuli were chosen each day based on the shape preferences of the neuron recorded (a preferred and non-preferred shape). B. The test stimulus was partially occluded by a field of 36 dots randomly positioned in a 9×9 square grid within the neuron’s RF. Task difficulty was titrated by varying the diameter of occluding dots; occlusion level was parameterized as the percentage of the RF area that remained unoccluded (% unoccluded area). Previously published in Kosai et al., (2014).

Figure 4

Results from an example curvature-tuned neuron during behavior. Same neuron as in Figure 1. A–B. Response PSTHs (σ = 10 ms) for the preferred (A) and non-preferred shapes (B) at different occlusion levels (colored lines) when presented as test stimuli. Responses to the preferred shape were strong when it was unoccluded (black; thin lines show SEM) and decreased with increasing occlusion level; the opposite occurred for responses to the non-preferred shape. C. Comparison of behavioral (gray) and neuronal (black) performance across occlusion level. Symbols indicate % correct performance at each occlusion level; lines are descriptive fits to the data. Neurometric curves were constructed based on responses in 50–350 ms counting window from test stimulus onset. Tick marks along the abscissa mark neurometric and psychometric thresholds (black and gray, respectively). Previously published in Kosai et al., 2014.

Figure 5

Results from a contrasting non-curvature tuned neuron. A. Same format as Figure 1. The neuron responded strongly to a few shapes, but its shape preferences were hard to describe in terms of local contour features. B–C. Responses to the preferred shape were strong when it was unoccluded and decreased rapidly with increasing occlusion level; responses to occluded shapes were weak, even for low occlusion levels (compare responses at 99 and 96% unoccluded area). Responses to the non-preferred shape were generally weak. D. Psychometric and neurometric curves were largely unmatched; behavioral performance was superior at most occlusion levels. Previously published in Kosai et al., 2014.

Figure 6

Relationship between model goodness of fit and threshold ratio. A. Threshold ratios versus the curvature model’s goodness of fit. For each neuron, we identified (using nonlinear least squares methods) the 2D Gaussian function in a shape space defined by angular position and boundary curvature, that best predicted neuronal responses to shape stimuli during passive fixation; the correlation between observed and predicted responses provided a measure of the goodness of fit. Neurons that were best fit by the model (curvature-tuned) had the lowest threshold ratios. Previously published in Kosai et al., 2014.

Figure 7

Time course of shape selectivity under partial occlusion. A. Data from an example neuron (same as in Fig. 1 & 4). A. Time course of selectivity for unoccluded shapes (black) and for shapes under different occlusion levels (colored lines). Selectivity for unoccluded shapes was strong and peaked soon after stimulus onset; selectivity weakened with increasing occlusion level and peaked later. Previously published in Kosai et al., 2014.