Multivoxel pattern selectivity for perceptually relevant binocular disparities in the human brain

Abstract

Processing of binocular disparity is thought to be widespread throughout cortex, highlighting its importance for perception and action. Yet the computations and functional roles underlying this activity across areas remain largely unknown. Here, we trace the neural representations mediating depth perception across human brain areas using multivariate analysis methods and high-resolution imaging. Presenting disparity-defined planes, we determine functional magnetic resonance imaging (fMRI) selectivity to near versus far depth positions. First, we test the perceptual relevance of this selectivity, comparing the pattern-based decoding of fMRI responses evoked by random dot stereograms that support depth perception (correlated RDS) with the decoding of stimuli containing disparities to which the perceptual system is blind (anticorrelated RDS). Preferential disparity selectivity for correlated stimuli in dorsal (visual and parietal) areas and higher ventral area LO (lateral occipital area) suggests encoding of perceptually relevant information, in contrast to early (V1, V2) and intermediate ventral (V3v, V4) visual cortical areas that show similar selectivity for both correlated and anticorrelated stimuli. Second, manipulating disparity parametrically, we show that dorsal areas encode the metric disparity structure of the viewed stimuli (i.e., disparity magnitude), whereas ventral area LO appears to represent depth position in a categorical manner (i.e., disparity sign). Our findings suggest that activity in both visual streams is commensurate with the use of disparity for depth perception but the neural computations may differ. Intriguingly, perceptually relevant responses in the dorsal stream are tuned to disparity content and emerge at a comparatively earlier stage than categorical representations for depth position in the ventral stream.

Figure 1.

Stimulus illustration and regions of interest. A, A representation of RDSs similar to those presented to observers. The illustration is designed for crossed-eye fusion. This can be achieved by first fixating the white space between the left and center images and then slowly crossing the eyes until the white-edged squares at the center of each image are lined up and two planes rendered in a correlated RDS are revealed (left plane near, right plane far as is illustrated by the right diagram in B). The same disparities are rendered in the fusion of the center and right images, but the luminance polarity of dots in the two eyes is reversed (a white dot in one eye matches a black dot in the other). This anticorrelated stimulus does not evoke a reliable disparity-defined depth. (Note that our subjects were not required to free fuse, and the fixation marker was considerably smaller.) B, A schematic representation of the disparity-defined depth structure of the stimuli. One of two depth configurations was presented on each trial: one in which the left plane was further and the right plane closer to the observer (diagram on left) or one in which the left plane was closer to the observer and the right plane further (cartoon on right). The planes to the left and right of fixation always had different signs (i.e., one near, one far) but the magnitude of the disparity was the same (i.e., disparity left = − disparity right). C, Regions of interest in one subject showing retinotopic areas, V3B/KO, hMT+/V5, LOC (LO and pFs subregions), and three-dimensional shape-related areas in the parietal cortex (VIPS, POIPS, and DIPS). Regions were defined using independent localizers (see Materials and Methods). Sulci are coded in darker gray than the gyri. Major sulci are labeled: STS, superior temporal sulcus; ITS, inferior temporal sulcus; CS, central sulcus.

Figure 2.

Prediction accuracy for near versus far discrimination. A, Mean prediction accuracy for the discrimination of crossed versus uncrossed disparity in different regions of interest. Dark bars represent performance of the classifier for fMRI responses evoked by correlated stimuli, and white bars represent performance for anticorrelated stimuli. The dotted horizontal lines depict chance performance (0.5 accuracy). Error bars depict the SEM across subjects (n = 8). B, Prediction accuracy expressed using an index to represent the preference for correlated stimuli. A value of 1 would indicate chance performance for responses evoked by anticorrelated stimuli and perfect classification performance for correlated stimuli. A value of 0 would indicate equal performance for correlated and anticorrelated stimuli. Error bars depict bootstrapped 95% confidence intervals for the index. *p < 0.05.

Figure 3.

fMRI pattern-based tuning curves. A, The proportion of predictions made to each of the stimulus disparity values in terms of the disparity difference between the viewed stimulus and the prediction. Each series corresponds to predictions made for activity evoked by a different disparity. The solid black line shows the best-fitting Gaussian to the data. Random predictions would correspond to a proportion of 0.167. B, The goodness-of-fit for the Gaussian model in each cortical region of interest. Error bars show 95% confidence intervals calculated from 1000 bootstrap samples. C, The full-width at half-maximum of the best-fitting Gaussian in each region of interest. Larger values correspond to a broader spread of predictions. Error bars show 95% confidence intervals calculated from 1000 bootstrap samples.

Figure 4.

Binary classifications: predicting the sign or magnitude of disparities. A, Prediction accuracy of classifiers distinguishing between two stimuli when those stimuli have either the same sign (red bars) or a different sign (blue bars). The magnitude of the difference in disparity between the two stimuli is illustrated by the bar saturation (less saturated colors indicate a 6 arcmin difference between stimuli, and bolder colors indicate a 12 arcmin difference). The dotted horizontal lines depict chance performance (0.5 accuracy). Error bars show the between-subjects SEM (n = 8). B, Prediction accuracy for a classification across the fixation plane as a function of the difference in disparity between the presented stimuli. The dotted horizontal lines depict chance performance (0.5 accuracy). Area LST/FST corresponds to the voxels that overlap between hMT+/V5 and LO. Areas labeled “MT & MST” correspond to the hMT+/V5 region excluding LST/FST. Error bars show the between-subjects SEM.

Figure 5.

Distribution of voxel biases. A, The distributions of t values yielded by contrasting crossed and uncrossed disparities in illustrative ROIs. The gray and black curves represent voxels in different hemispheres that have access to stimuli in the left and right visual fields, respectively. Vertical lines indicate the means of the distributions. Differences in the mean value represent the presence of a univariate signal. A preference for crossed disparities is shown by a rightward shift in the gray distribution and a leftward shift in the black distribution. Curves for all areas are shown in supplemental Figure S6 (available at www.jneurosci.org as supplemental material). B, The difference in the mean of the t value distributions for the left and right hemispheres for all regions. Values significantly greater than 0 are marked with an asterisk. Error bars are SEM across subjects (n = 8). C, The difference in the skew of the t value distributions. Values significantly greater than 0 are marked with an asterisk. Error bars are SEM across subjects (n = 8). D, The variance of the t value distributions. A higher variance indicates a wider spread of t values (i.e., a higher proportion of voxels with a strong preference for crossed or uncrossed disparities). Error bars show SEM across subjects (n = 8). E, The difference in the saturation rates for pattern size by prediction accuracy curves when voxels are ordered by their significance compared with being randomly sampled. A larger value indicates a larger difference in saturation rate and provides an indication that the prescribed voxel ordering is more critical. Error bars show SEM across subjects (n = 8).