The representation of object viewpoint in human visual cortex

Abstract

Understanding the nature of object representations in the human brain is critical for understanding the neural basis of invariant object recognition. However, the degree to which object representations are sensitive to object viewpoint is unknown. Using fMRI we employed a parametric approach to examine the sensitivity to object view as a function of rotation (0 degrees-180 degrees ), category (animal/vehicle) and fMRI-adaptation paradigm (short or long-lagged). For both categories and fMRI-adaptation paradigms, object-selective regions recovered from adaptation when a rotated view of an object was shown after adaptation to a specific view of that object, suggesting that representations are sensitive to object rotation. However, we found evidence for differential representations across categories and ventral stream regions. Rotation cross-adaptation was larger for animals than vehicles, suggesting higher sensitivity to vehicle than animal rotation, and was largest in the left fusiform/occipito-temporal sulcus (pFUS/OTS), suggesting that this region has low sensitivity to rotation. Moreover, right pFUS/OTS and FFA responded more strongly to front than back views of animals (without adaptation) and rotation cross-adaptation depended both on the level of rotation and the adapting view. This result suggests a prevalence of neurons that prefer frontal views of animals in fusiform regions. Using a computational model of view-tuned neurons, we demonstrate that differential neural view tuning widths and relative distributions of neural-tuned populations in fMRI voxels can explain the fMRI results. Overall, our findings underscore the utility of parametric approaches for studying the neural basis of object invariance and suggest that there is no complete invariance to object view in the human ventral stream.

Figure 1. Stimuli and Experimental design

a) Examples of animal and vehicle stimuli used in the experiments. b) Examples of experimental conditions in adaptation experiments: left column depicts the adapting stimulus, right the test stimulus. In the short-lagged adaptation paradigm, adapting and test stimuli appeared in succession. In the long-lagged paradigm, the test stimulus appeared a few minutes after the adapting stimulus with many intervening stimuli.

Figure 2. Object selective activations

Object selective activations for a representative subject shown on the inflated (top) and flattened cortex (bottom). Activations for animals > scrambled (P < .00001, voxel level, yellow) and vehicles > scrambled (P < .00001, voxel level) during the localizer experiment are superimposed. Regions that activated to both contrasts are shown in green. The FFA was defined as a region in the fusiform gyrus (or occipito-temporal sulcus) that responded more to faces than cars and novel objects (P < .001, voxel level). Red, blue and green lines denote visual meridians from the retinotopy experiment. MT is marked as a region in the posterior bank of the inferior temporal sulcus that responded more strongly for moving than stationary low contrast rings (P < .0001).

Figure 3. Behavioral Responses

(a) Mean response time (RT) and accuracy (proportion correct) averaged across 8 subjects who participated in Experiment 1. Error bars indicate standard error of the mean (SEM). (b) Behavioral responses for the test stimulus (second stimulus in each pair) during the short-lagged fMRI-adaptation experiment averaged across 7 subjects. Error bars: SEM. (c) Behavioral responses for the test stimulus during the long-lagged fMRI-adaptation experiment averaged across 8 subjects. Error bars indicate SEM.

Figure 4. BOLD responses to object views across object, and face-selective cortex

Mean fMRI responses to animal (solid) and vehicle (dashed) views relative to a scrambled baseline. Data are averaged across 8 subjects. Error bars indicate SEM.

Figure 5. LO responses for vehicles during short and long-lagged fMRI-A experiments

Panel a) shows results of short-lagged adaptation study, while panel b) shows long-lagged results. Blue: Adapting stimulus was presented in the front view. Red: Adapting stimulus was presented in the back view. Diamonds: responses to novel objects shown in the same view as adapting objects. Open circles: significant adaptation (lower than novel objects, P < 0.05, paired t-test across subjects). Responses are plotted relative to a blank (with fixation) baseline. Error bars indicate averaged SEM.

Figure 6. LO responses for animals during short and long-lagged fMRI-A experiments

Panel a) shows results of short-lagged adaptation study, while panel b) shows long-lagged results. Blue: Adapting stimulus was presented in the front view. Red: Adapting stimulus was presented in the back view. Diamonds: responses to novel objects shown in the same view as adapting objects. Open circles: significant adaptation (lower than novel objects, P < 0.05, paired t-test across subjects). Responses are plotted relative to a blank (with fixation) baseline. Error bars indicate averaged SEM.

Figure 7. V1 responses across object viewpoint during short and long-lagged fMRI-A experiments

Responses are plotted as a function of the object viewpoint with percentage signal change calculated relative to a blank (with fixation) baseline. Blue: Adapting stimulus was presented in the front view. Red: Adapting stimulus was presented in the back view. Diamonds: responses to novel objects shown in the same view as adapting objects. Error bars indicate averaged SEM. V1 revealed no significant adaptation, but preference to object views that had the largest horizontal extent (see Figure 8a).

Figure 8. Stimulus size, pixel-wise dissimilarity & behavioral similarity

a) Mean visual angle of stimuli measured horizontally (left) and vertically (right) for each category and viewpoint. Error bars represent one standard deviation across images. b) Average pixel-wise image dissimilarity between all adapting and test stimuli used in the experiments. Blue: versus the 15° view. Red: versus the 195° view. Rotated versions of the same object (filled circles) are more dissimilar (t-test, P < .05) than two different objects in the same view (Different condition). c) Subjects’ discrimination performance (d′) for the same-different discrimination task as a function of rotation.

Figure 9. Mixture of View-Dependent Neural Populations Model

Left: Schematic illustration of the view-tuning and distribution of neural populations tuned to different views in a voxel. Each bar represents the relative proportion of neurons in a voxel tuned to a view, and each is color coded by the preferred view. Right: result of computational implementation of the model illustrating predicted BOLD responses during fMRI-A experiments. Diamonds: responses without adaptation; Lines: response after adaptation with a front view (blue line) or back view (red line). Across columns the view tuning width varies, across rows the distribution of neural populations preferring specific views varies. a) Mixture of view-dependent neural populations which are equally distributed in a voxel. This model predicts a similar pattern of recovery from adaptation regardless of the adapting view. b) Mixture of view-dependent neural populations in a voxel with a higher proportion of neurons that prefer the front view. This model predicts higher BOLD responses to frontal views of animals without adaptation and a different pattern of rotation cross-adaptation during fMRI-A experiments.