Multiple objects evoke fluctuating responses in several regions of the visual pathway

Abstract

How neural representations preserve information about multiple stimuli is mysterious. Because tuning of individual neurons is coarse (e.g., visual receptive field diameters can exceed perceptual resolution), the populations of neurons potentially responsive to each individual stimulus can overlap, raising the question of how information about each item might be segregated and preserved in the population. We recently reported evidence for a potential solution to this problem: when two stimuli were present, some neurons in the macaque visual cortical areas V1 and V4 exhibited fluctuating firing patterns, as if they responded to only one individual stimulus at a time (Jun et al., 2022). However, whether such an information encoding strategy is ubiquitous in the visual pathway and thus could constitute a general phenomenon remains unknown. Here, we provide new evidence that such fluctuating activity is also evoked by multiple stimuli in visual areas responsible for processing visual motion (middle temporal visual area, MT), and faces (middle fundus and anterolateral face patches in inferotemporal cortex - areas MF and AL), thus extending the scope of circumstances in which fluctuating activity is observed. Furthermore, consistent with our previous results in the early visual area V1, MT exhibits fluctuations between the representations of two stimuli when these form distinguishable objects but not when they fuse into one perceived object, suggesting that fluctuating activity patterns may underlie visual object formation. Taken together, these findings point toward an updated model of how the brain preserves sensory information about multiple stimuli for subsequent processing and behavioral action.

Keywords: figure ground segregation; multiplexing; neural code; neural representation; neuroscience; object vision; rhesus macaque; visual system.

Figure 1.. Description of datasets.

(a) Middle temporal area (MT) gratings datasets from Ruff and Cohen, 2016b and Ruff et al., 2016a. (a.1). For the adjacent gratings, the two stimuli were placed side by side and roughly within the MT neuron’s receptive field. The superimposed gratings were larger and each was placed at the same position, also within the MT neuron’s receptive field. (a.2) Monkeys performed an attention task during the presentation of the adjacent gratings; only trials in which attention was successfully directed to the stimulus in the ipsilateral hemifield were included in this study. (a.3) Monkeys performed a fixation task during presentation of the superimposed gratings. (b) MT random dot patch dataset from Li et al., 2016. Patches of moving dots were presented within apertures placed within the receptive field of an MT neuron. Attention was directed to the fixation target for the trials included in the present study. (c) Face patch stimulus conditions (Ebihara, 2015). (c.1) Monkeys performed a fixation task and stimuli were presented either at the center of the receptive field or from one of eight locations surrounding it. (c.2) ‘Preferred’ faces were faces that evoked a strong response, and were presented in the center of the receptive field. Non-preferred faces or non-face objects were presented at one of the other locations. Combinations of stimuli involved the preferred face in the center and one of the non-preferred stimuli at one of the adjacent locations. (c.3) Additional examples of stimuli used in the face patch datasets.

Figure 1—figure supplement 1.. Properties of the datasets and application of exclusion criteria.

(a) Average A, B, and AB trial counts for each included triplet (after Poisson, trial count, and response separation screening). (b) The distribution of variance-to-mean ratios, or Fano factors, on A- and B-alone trials across the different datasets.Results shown involve the average A and B Fano factors for a given triplet (computed after the screening for minimum trial count and A and B response separation). (c) A and B response separations, computed as an index of the difference between A and B responses as a fraction of their sum, for included triplets (after Poisson, trial count, and response separation screening). Note that the middle temporal area (MT) plaid dataset differs from the other 4 datasets in that many of the included triplets have much smaller A and B response separations than those that are included for other datasets. This is likely due to the larger numbers of trials providing greater confidence that the A and B responses did indeed differ at smaller difference levels. (d) Average ‘winning probability’ for the ‘mixture’ model (see Materials and methods: Data analysis) as a function of the response separation. Average ‘mixture’ winning probability was higher for all four datasets involving distinguishable stimuli (MT adjacent gratings and dot patches, the face patch datasets) than for the MT plaid dataset, across all levels of A vs. B response separation.

Figure 2.. Statistical comparisons to be made across datasets.

We assign each trial into a ‘triplet’ consisting of trials in which stimulus A (red), stimulus B (blue), or both stimulus A and B (AB, black) were presented. (a) Within each triplet, spike count distributions are compared to determine how the AB distribution relates to the A and B distributions. (b) The AB spike count distribution may be: a mixture of the A and B distributions, suggesting fluctuations between the responses to the two stimuli; an intermediate rate between the A and B distributions, suggesting averaging (or fluctuations on a faster time scale than the analysis period); a rate that matches either the single A or B distribution, suggesting a winner-take-all strategy; or a rate outside the range bounded by the A and B distributions, suggesting addition or subtraction of the responses. Spike counts are assumed to be drawn from Poisson distributions, with a weighting parameter alpha defining the relative contribution of the individual A and B distributions in ‘mixtures’ and ‘intermediates’.

Figure 3.. Example individual unit responses and population results for middle temporal area (MT).

(a) An example unit’s response to two simultaneous adjacent gratings (AB, black solid lines) in comparison to the mean response to each grating individually (A and B, red and blue dotted lines). (b) A different example unit’s response to simultaneous dot patches. In both (a) and (b), gray bars indicate the raw distributions. The black lines indicate a smoothed fit for display purposes; this fit was generated by convolution with a 3-point sliding window of which the middle point is weighted twice as much as either of the two outer points followed by cubic spline interpolation. Spike counts were computed for a window 200 ms in duration (see Materials and Methods); multiply x-axis by 5 to convert to spike rates in Hz. (c) Number of triplets classified into each category (as shown in Figure 2) when responses to two simultaneous gratings (dark blue) or dot patches (light blue) are recorded in MT. Only triplets for which the winning model garnered a posterior probability of 0.67 or greater are shown. (d) Same as (c) but for fused objects, created by overlaying two gratings to form a plaid. (e) Comparison of (c) and (d) showing that fluctuating (mixture) responses are only present when two objects are distinct, and not when two objects are fused into one. Results in this figure only include triplet conditions for which the average variance-to-mean ratio (Fano factor) of activity on the A- and B-only trials was less than 3. See Figure 4—figure supplement 1 for corresponding findings using stricter inclusion criteria.

Figure 4.. Example individual unit responses and population results for IT face patches middle fundus (MF) and AL.

(a) An example single MF unit’s response to two simultaneous faces (AB, black solid lines) in comparison to the mean response to each face individually (A and B, red and blue dotted lines). (b) Same as (a) but in response to one face and one object. (c) Same as (a) but recorded in AL. (d) Same as (b) but recorded in AL. In (a) through (d), distributions were smoothed for display purposes via convolution with a 3-point sliding window of which the middle point is weighted twice as much as either of the two outer points, followed by cubic spline interpolation. Gray bars show the raw spike counts for the 200 ms spike counting windows (multiply x-axis scale by 5 to convert to spike rates in Hz). (e) Number of triplets classified into each category (as shown in Figure 2) when responses to a face–face pair (dark brown) or a face–object pair (mustard color) are recorded in face patch MF. Only triplets for which the winning model garnered a posterior probability of 0.67 or greater are shown. (f) Same as (e) but recorded in face patch AL. (g) Comparison of (e) and (f) showing that more conditions display a fluctuating (mixture) pattern in AL than in MF. In MF, there were more ‘mixtures’ among face–face pairs than among face–object pairs. Results in this figure only include triplet conditions for which the average variance-to-mean ratio (Fano factor) of activity on the A- and B-only trials was less than 3. See Figure 4—figure supplement 1 for corresponding findings using stricter inclusion criteria.

Figure 4—figure supplement 1.. Same as population results in Figures 3 and 4, but using a Fano factor criterion of 2 instead of 3.

All trends in the data are maintained compared to the comparisons with a criterion of 3, other than the loss of significance between face–face and face–object comparisons in middle fundus (MF).