Figure-ground organization and the emergence of proto-objects in the visual cortex

Abstract

A long history of studies of perception has shown that the visual system organizes the incoming information early on, interpreting the 2D image in terms of a 3D world and producing a structure that provides perceptual continuity and enables object-based attention. Recordings from monkey visual cortex show that many neurons, especially in area V2, are selective for border ownership. These neurons are edge selective and have ordinary classical receptive fields (CRF), but in addition their responses are modulated (enhanced or suppressed) depending on the location of a 'figure' relative to the edge in their receptive field. Each neuron has a fixed preference for location on one side or the other. This selectivity is derived from the image context far beyond the CRF. This paper reviews evidence indicating that border ownership selectivity reflects the formation of early object representations ('proto-objects'). The evidence includes experiments showing (1) reversal of border ownership signals with change of perceived object structure, (2) border ownership specific enhancement of responses in object-based selective attention, (3) persistence of border ownership signals in accordance with continuity of object perception, and (4) remapping of border ownership signals across saccades and object movements. Findings 1 and 2 can be explained by hypothetical grouping circuits that sum contour feature signals in search of objectness, and, via recurrent projections, enhance the corresponding low-level feature signals. Findings 3 and 4 might be explained by assuming that the activity of grouping circuits persists and can be remapped. Grouping, persistence, and remapping are fundamental operations of vision. Finding these operations manifest in low-level visual areas challenges traditional views of visual processing. New computational models need to be developed for a comprehensive understanding of the function of the visual cortex.

Keywords: attention; contour grouping; neural mechanism; object files; object permanence; perceptual organization; single-cell recording; visual cortex.

FIGURE 1

Border ownership selectivity. (A–C) Responses of an example V2 neuron. Rows of tics in (A,B) represent repeated responses to the stimuli shown on the left. (A) The responses to the right-hand edge of a bright square (top) are much stronger than the responses to the left-hand edge of a dark square (bottom) despite stimuli being identical in the receptive field (green ellipse, not part of display): the neuron is sensitive to the image context. (B) Same as (A), but contrast reversed. Again, the cell responds more strongly when the square is to the left of the receptive field: the neuron prefers ‘left’ border ownership. (C) Average time course of the neuron’s firing rate for left (red) and right (blue) border ownership. Note divergence of curves right after response onset (68 ms after stimulus onset in this neuron). (D–F) Demonstration of the range of context integration. A V2 Neuron was examined with edges as in (A), but with contour-defined squares where luminance variations were confined to a narrow seam at the contours (D). The contours were broken up into eight fragments (four edges and four corners); one edge was placed over the receptive field, while the seven ‘contextual’ fragments were presented randomly in all possible combinations. (E) The influence of each of the contextual fragments on the responses, as determined by regression analysis. Colored shading represents the regression coefficients, red indicating enhancement, blue, suppression. Results are shown separately for left and right locations of square, and for two sizes of square (3° and 8° visual angle). The small gray specks on the test edge show the map of the neuron’s classical receptive field determined with flashing bars. The small size of this near-foveal receptive field contrasts with the large range of context integration: nearly all contour fragments to the left of the receptive field enhanced the responses, whereas contour fragments to the right suppressed them. (F) The result of presenting the same contextual fragments as in (E), but without the edge in the receptive field: the contextual fragments alone produced no response. (Modified from Zhang and von der Heydt, 2010.)

FIGURE 2

Illustration of the face recognition experiment of Nakayama et al. (1989). When the face is in back, it is easy to identify, but when the face strips are in front, the system assigns their borders to the strips, thus grouping the face information with the horizontal borders, and face identification becomes difficult. The illustration uses pictorial tools to depict depth, in the experimental study depth was defined by stereoscopic presentation. (Reproduced with permission from Nakayama and Shimojo, 2009.)

FIGURE 3

Border ownership signals correlate with perceptual organization. (A, Top) This configuration of two light and two dark squares is generally perceived as a pair of crossed bars in transparent overlay, or as a light bar with a shadow on it, or a dark bar crossed by a beam of light. However, when the corners of the squares are rounded off, only the individual squares are perceived (A, bottom). (B) Curves show the average border ownership signals (difference between preferred and non-preferred side responses) for the three conditions shown at the top, where a green ellipse marks the location and approximate size of receptive fields (not part of display). Note the reversals of perceived border ownership of the marked edge between a (owned right), b (owned left), and c (owned right). The neural border ownership signals reverse sign accordingly. (Modified from Qiu and von der Heydt, 2007.)

FIGURE 4

The influence of attention on the responses of a typical border ownership selective neuron. The receptive field of the neuron (green ellipse, not part of display) was placed on the border between two overlapping figures. In one configuration (rows 1 and 2) this ‘occluding edge’ is owned by the square on the right, in the other configuration (rows 3 and 4) it is owned by the square on the left. Attention was controlled by having the monkey perform a shape discrimination task with one of the figures (according to preceding instruction). The attended figure is marked here by a yellow asterisk (not part of display). ‘Left’ border ownership produced stronger responses than ‘right’ border ownership in both attention conditions. Interestingly, the attention effect was also asymmetric. Attention on the left figure enhanced the responses compared to attention on the right figure, irrespective of border ownership (compare rows 1 and 2 and rows 3 and 4). This asymmetry of the attention influence was systematic across the population: attention enhanced responses on the preferred border ownership side, relative to attention on the non-preferred side. (Modified from Qiu et al., 2007.)

FIGURE 5

A neural grouping model. (A) Edge-selective neurons at the Feature level activate two populations of border ownership neurons (B-cells), one for each side of the edge. These neurons send signals to specific processing areas such as inferotemporal cortex, but they have also collateral projections to ‘Grouping cells’ (G-cells) which sum their signals, collecting edge signals in co-circular configuration. G-cells, by feedback, enhance the responses of the same B-cells (Inset, dashed arrows). Each B-cell is exclusively connected to G-cells on one side; thus, the enhancement makes the B-cell border ownership selective for that side. This is indicated by arrows on the receptive fields of the corresponding color. G-cells also sum responses of end-stopped cells (black) signaling local occlusion features like T-junctions which are indicators of border ownership. Assuming that top-down attention targets the G-cells, the model also provides a parsimonious explanation of object-based attention. (B) The model predicts that two B-cells that are connected to the same G-cell (red dashed line) will exhibit increased synchrony of firing when both are stimulated by the same object (bound) compared to activation by different objects (unbound), whereas B-cells that do not share common G-cells (black dashed lines) will show no increased synchrony. A recent study confirmed exactly these predictions (Martin and von der Heydt, 2015). (Modified from Craft et al., 2007.)

FIGURE 6

Object continuity and persistence of grouping. (Top left) Two kinds of movement displays were used. In a, an inverted L shape initially occluded part of a dark square (top). The square then moved smoothly to a new position, resulting in the configuration shown below, which is typically perceived as a dark square occluding a white rectangle. Thus, the perceived direction of occlusion (border ownership) changes after the termination of movement. In sequence b, the dark square initially occludes the white shape and the latter then moves to a new position, resulting in the same final configuration as in a. (Bottom left) Average border ownership signals (difference between preferred and non-preferred side responses) for the two display sequences. In condition b, the border ownership signal shoots up shortly after the figures are presented (left vertical dotted line), reaching a peak and then decaying slowly. In condition a, the border ownership signal first turns negative, indicating ‘left’ border ownership, and then slowly drifts toward positive values. The vertical dotted line at time 0 indicates the end of the movement. At this time, the displays in the two conditions become identical, the only difference being the history of events. However, the border ownership signals remain different for at least 1500 ms. (Right) The Schematic illustration of the events in terms of the grouping cell model. In condition b, none of the borders changes assignment. During the movement, dynamic cues (accretion/deletion) as well as static cues (T-junctions, compact shape of square) define ‘right’ border ownership at the occluding contour. After the termination of movement, the static cues continue to activate the G-cell corresponding to the square (blue), keeping the occluding contour assigned. In condition a, dynamic cues and T-junctions initially indicate the left figure to be in front, activating the red G-cell most strongly. But when the movement terminates, the dynamic cues disappear and the static cues now indicate ‘right’ ownership for the occluding contour. However, the experiment showed that the border ownership neurons kept signaling ‘left’ for about 800 ms, indicating that the elevated activity in the G-cell persisted (red dotted circle). The persistence of border assignment despite the reversal figure–ground cues indicates that the grouping mechanisms have memory (see Discussion). (Modified from O’Herron and von der Heydt, 2011.)