Natural scene statistics account for the representation of scene categories in human visual cortex

Abstract

During natural vision, humans categorize the scenes they encounter: an office, the beach, and so on. These categories are informed by knowledge of the way that objects co-occur in natural scenes. How does the human brain aggregate information about objects to represent scene categories? To explore this issue, we used statistical learning methods to learn categories that objectively capture the co-occurrence statistics of objects in a large collection of natural scenes. Using the learned categories, we modeled fMRI brain signals evoked in human subjects when viewing images of scenes. We find that evoked activity across much of anterior visual cortex is explained by the learned categories. Furthermore, a decoder based on these scene categories accurately predicts the categories and objects comprising novel scenes from brain activity evoked by those scenes. These results suggest that the human brain represents scene categories that capture the co-occurrence statistics of objects in the world.

Figure 1. Overview of Analyses

(A) Learning Database. We compiled a large database of labeled natural scenes. All objects in each of the scenes were labeled by naïve participants. See also Supplemental Figure S2. (B) Scene Categories Learned by LDA. LDA was used to learn scene categories that best capture the co-occurrence statistics of objects in the learning database. LDA defines each scene category as a list of probabilities, where each probability is the likelihood that any particular object within a fixed vocabulary will occur in a scene. Lists of probable objects for four example scene categories learned by LDA are shown on the right. Each list of object labels corresponds to a distinct scene category; within each list saturation indicates an object’s probability of occurrence. The experimenters, not the LDA algorithm, assigned intuitive category names in quotes. Once a set of categories is learned, LDA can also be used to infer the probability that a new scene belongs to each of the learned categories, conditioned on the objects in the new scene. See also Supplemental Figure S2. (C) Voxel-wise Encoding Model Analysis. Voxel-wise encoding models were constructed to predict BOLD responses to stimulus scenes presented during an fMRI experiment. Blue represents inputs to the encoding model, green represents intermediate model steps, and red represents model predictions. To generate predictions, the labels associated with each stimulus scene (blue box) are passed to the LDA algorithm (dashed green oval). LDA is used to infer from these labels the probability that the stimulus scene belongs to each of the learned categories (solid green oval). In this example the stimulus scene depicts a plate of fish so the scene categories “Dining” and “Aquatic” are highly probable (indicated by label saturation), while the category “Roadway” is much less probable. These probabilities are then transformed into a predicted BOLD response (red diamond) by a set of linear model weights (green hexagon). Model weights were fit independently for each voxel using a regularized linear regression procedure applied to the responses evoked by a set of training stimuli. (D) Decoding Model Analysis. A decoder was constructed for each subject that uses BOLD signals evoked by a viewed stimulus scene to predict the probability that the scene belongs to each of a set of learned scene categories. Blue represents inputs to the decoder, green represents intermediate model steps, and red represents decoder predictions. To generate a set of category probability predictions for a scene (red diamond), evoked population voxel responses (blue box) are mapped onto the category probabilities by a set of multinomial model weights (green hexagon). Predicted scene category probabilities were then used in conjunction with the LDA algorithm to infer the probabilities that specific objects occurred in the viewed scene (red oval). The decoder weights were fit using regularized multinomial regression applied to the scene category probabilities inferred for a set of training stimuli using LDA and the responses to those stimuli.

Figure 2. Identifying the Best Scene Categories for Modeling Data Across Subjects

(A) Encoding model performance across a range of settings for the specified number of distinct categories learned using LDA (y-axis) and vocabulary size (x-axis). Each pixel corresponds to one of the candidate scene categories learned by LDA when applied to the learning database. The color of each pixel represents the relative amount of cortical territory across subjects that is accurately predicted by encoding models based on a specific setting for the number of individual categories and vocabulary size. The number of individual categories was incremented from 2 to 40. The object vocabulary was varied from the 25 most frequent to the 950 most frequent objects in the learning database. The red dot identifies the number of individual categories and vocabulary size that produce accurate predictions for the largest amount of cortical territory across subjects. For individual results see Supplemental Figure S3. (B) Ten examples taken from the 20 best scene categories identified across subjects (corresponding to the red dot in (A)). The seven most probable objects for each category are shown. Format is the same as in Figure 1B. See Supplemental Figures S4–S5 for interpretation of all 20 categories.

Figure 3. Scene Categories Learned from Natural Scenes are Encoded in Many Anterior Visual ROIs

(A) Encoding model prediction accuracies are mapped onto the left (LH) and right (RH) cortical surfaces of one representative subject (S1). Gray indicates areas outside of the scan boundary. Bright locations indicate voxels that are accurately predicted by the corresponding encoding model (prediction accuracy at two levels of statistical significance—p < 0.01 (r = 0.21) and p < 0.001 (r = 0.28)—are highlighted on the colorbar). ROIs identified in separate retinotopy and functional localizer experiments are outlined in white. The bright regions overlap with a number of the ROIs in anterior visual cortex. These ROIs are associated with representing various high-level visual features. However, the activity of voxels in retinotopic visual areas (V1, V2, V3, V4, V3a, V3b) are not predicted accurately by the encoding models. Prediction accuracy was calculated on responses from a separate validation set of stimuli not used to estimate the model. ROI Abbreviations: Retinotopic Visual Areas 1–4 (V1–V4); Para-hippocampal Place Area (PPA); FFA, Fusiform Face Area (PPA); Extrastriate Body Area (EBA); Occipital Face Area (OFA); Retrosplenial Cortex (RSC); Transverse Occipital Sulcus (TOS). Center key: A=anterior; P=posterior, S=superior; I=inferior. For remaining subjects’ data, see Supplemental Figure S6. (B) Each bar indicates the average proportion of voxel response variance in an ROI that is explained by voxel-wise encoding models estimated for a single subject. Bar colors distinguish individual subjects. Error bars are standard error from the mean. For all anterior visual ROIs and for all subjects, encoding models based on scene categories learned from natural scenes explain a significant proportion of voxel response variance (p < 0.01, indicated by red lines). (C) The average encoding model weights for voxels within distinct functional ROIs. Averages are calculated across all voxels located within the boundaries of an ROI, and across subjects. Each row displays the average weights for the scene category listed on the left margin. Each column distinguishes average weights for individual ROIs. The color of each pixel represents the positive (red) or negative (blue) average ROI weight for the corresponding category. The size of each pixel is inversely proportional to the magnitude of the standard error of the mean estimate; larger pixels indicate selectivity estimates with greater confidence. Standard error scaling is according to the data within an ROI (column). ROI tuning is generally consistent with previous findings. However, tuning also appears to be more complex than indicated by conventional ROI-based analyses For individual subjects’ data, see Supplemental Figure S7; see also Supplemental Figures S8–S15.

Figure 4. Scene Categories and Objects Decoded from Evoked BOLD Activity

(A) Examples of scene category and object probabilities decoded from evoked BOLD activity. Blue boxes (columns 1 and 4) display novel stimulus scenes observed by subjects S1 (top row) through S4 (bottom row). Each red box (columns 2 and 5) encloses the top category probabilities predicted by the decoder for the corresponding scene to the left. The saturation of each category name within the red boxes represents the predicted probability that the observed scene belongs to the corresponding category. Black boxes (columns 3 and 6) enclose the objects with the highest estimated probability of occurring in the observed scene to the left. The saturation of each label within the black boxes represents the estimated probability of the corresponding object occurring in the scene. See also Supplemental Figures S16–S19. (B) Decoding accuracy for predicted category probabilities. Category decoding accuracy for a scene is the correlation coefficientbetween the category probabilities predicted by the decoder and the category probabilities inferred directly using LDA. Category probabilities were decoded for 126 novel scenes. Each plot shows the (horizontally mirrored) histogram of decoding accuracies for a single subject. Median decoding accuracy and 95% confidence interval (CI) calculated across all decoded scenes is represented by black cross-hairs overlayed on each plot. For subjects S1–S4, median decoding accuracy was 0.72 (CI : [0.62, 0.78]), 0.68 (CI: [0.53, 0.80]), 0.65 (CI: [0.55, 0.72]), 0.80 (CI: [0.72, 0.85]), respectively. For a given image, decoding accuracy greater than 0.58 was considered statistically significant (p < 0.01), and is indicated by the red line. A large majority of the decoded scenes are statistically significant, including all examples shown in (A). (C) Decoding accuracy for predicted object probabilities. Object decoding accuracy is the ratio of the likelihood of the objects labeled in each scene given the decoded category probabilities, to the likelihood of the labeled objects in each scene if all were selected with equal probability (chance). A likelihood ratio greater than one (red line) indicates that the objects in a scene are better predicted by the decoded object probabilities than by selecting objects randomly. Each plot shows the (horizontally mirrored) histogram of likelihood ratios for a single subject. Median likelihood ratios and 95% CI are represented by the black cross-hairs. For subjects S1–S4, the median likelihood ratio was 1.67 (CI: [1.57, 1.76]), 1.66 (CI: [1.52, 1.72]), 1.62 (CI: [1.45, 1.78]), 1.66 (CI: [1.56, 1.78]) for subjects S1–S4, respectively.