Associative Processing Is Inherent in Scene Perception

Abstract

How are complex visual entities such as scenes represented in the human brain? More concretely, along what visual and semantic dimensions are scenes encoded in memory? One hypothesis is that global spatial properties provide a basis for categorizing the neural response patterns arising from scenes. In contrast, non-spatial properties, such as single objects, also account for variance in neural responses. The list of critical scene dimensions has continued to grow--sometimes in a contradictory manner--coming to encompass properties such as geometric layout, big/small, crowded/sparse, and three-dimensionality. We demonstrate that these dimensions may be better understood within the more general framework of associative properties. That is, across both the perceptual and semantic domains, features of scene representations are related to one another through learned associations. Critically, the components of such associations are consistent with the dimensions that are typically invoked to account for scene understanding and its neural bases. Using fMRI, we show that non-scene stimuli displaying novel associations across identities or locations recruit putatively scene-selective regions of the human brain (the parahippocampal/lingual region, the retrosplenial complex, and the transverse occipital sulcus/occipital place area). Moreover, we find that the voxel-wise neural patterns arising from these associations are significantly correlated with the neural patterns arising from everyday scenes providing critical evidence whether the same encoding principals underlie both types of processing. These neuroimaging results provide evidence for the hypothesis that the neural representation of scenes is better understood within the broader theoretical framework of associative processing. In addition, the results demonstrate a division of labor that arises across scene-selective regions when processing associations and scenes providing better understanding of the functional roles of each region within the cortical network that mediates scene processing.

Fig 1. Experimental Conditions.

Examples of the training phase. Identity (ID)–the same three shapes presented together in random configurations; Spatial (SP)–three random shapes presented in the same configuration; Spatial-Identity (SPID)–the same three shapes always presented together in the same configuration; No Association (NA)–three random shapes presented in random configurations. Each stimulus was repeated thirty times in the training phase.

Fig 2

A) Whole brain analysis comparing BOLD activity elicited for the associative shapes (SPID vs. NA, neon green) with the activity elicited for the scenes (Scenes vs. Objects and Scrambled, teal). Both contrasts revealed regions of the brain with overlapping significant differential activity and particularly within the PPA, RSC, and OPA. B) Region of interest analysis for the PPA, RSC, and OPA. Bar graphs show the activity that was greater the control NA condition; negative values would indicate that NA > associative conditions, which was not found. B–bottom) a subset of the participants who performed above chance in all three associative conditions.

Fig 3. LH PPA posterior (spatial) to anterior (non-spatial) gradient of information processing.

A) the correlation in the pattern of activity across voxels in five bins going from the posterior to anterior regions of the PPA for unthresholded t maps of SP vs. NA with Scenes vs. Objects+Scrambled in purple and ID vs. NA with Scenes vs. Objects+Scrambled in yellow. B) Taking the correlation of these contrasts from one extreme end and adding more PPA until the entire PPA is surveyed.