Scene Perception and Visuospatial Memory Converge at the Anterior Edge of Visually Responsive Cortex

Abstract

To fluidly engage with the world, our brains must simultaneously represent both the scene in front of us and our memory of the immediate surrounding environment (i.e., local visuospatial context). How does the brain's functional architecture enable sensory and mnemonic representations to closely interface while also avoiding sensory-mnemonic interference? Here, we asked this question using first-person, head-mounted virtual reality and fMRI. Using virtual reality, human participants of both sexes learned a set of immersive, real-world visuospatial environments in which we systematically manipulated the extent of visuospatial context associated with a scene image in memory across three learning conditions, spanning from a single FOV to a city street. We used individualized, within-subject fMRI to determine which brain areas support memory of the visuospatial context associated with a scene during recall (Experiment 1) and recognition (Experiment 2). Across the whole brain, activity in three patches of cortex was modulated by the amount of known visuospatial context, each located immediately anterior to one of the three scene perception areas of high-level visual cortex. Individual subject analyses revealed that these anterior patches corresponded to three functionally defined place memory areas, which selectively respond when visually recalling personally familiar places. In addition to showing activity levels that were modulated by the amount of visuospatial context, multivariate analyses showed that these anterior areas represented the identity of the specific environment being recalled. Together, these results suggest a convergence zone for scene perception and memory of the local visuospatial context at the anterior edge of high-level visual cortex.SIGNIFICANCE STATEMENT As we move through the world, the visual scene around us is integrated with our memory of the wider visuospatial context. Here, we sought to understand how the functional architecture of the brain enables coexisting representations of the current visual scene and memory of the surrounding environment. Using a combination of immersive virtual reality and fMRI, we show that memory of visuospatial context outside the current FOV is represented in a distinct set of brain areas immediately anterior and adjacent to the perceptually oriented scene-selective areas of high-level visual cortex. This functional architecture would allow efficient interaction between immediately adjacent mnemonic and perceptual areas while also minimizing interference between mnemonic and perceptual representations.

Keywords: fMRI; memory; scene perception; virtual reality; visual cortex; visuospatial context.

Figure 1.

Experimental design to test processing of visuospatial context associated with a real-world scene. The experiment took place across three sessions: one study session and two fMRI sessions. These sessions took place on 3 separate days. A, In the VR study session, participants studied 20 real-world visual scenes with varying amounts of visuospatial context in head-mounted VR. Each scene was associated with one of three levels of spatial visuospatial context: single images (45° visible, 315° occluded), panoramas (270° visible, 90° occluded), and streets (three contiguous, navigable 360° photospheres). During the two fMRI sessions, we tested the neural response during recall (B) and recognition (C) of the learned scenes using fMRI.

Figure 2.

Aerial view of three study conditions. The “key scene views” were used as the stimuli for Experiment 2. The key scene view was always looking “down a street” to control for visual information across the conditions.

Figure 3.

Regions in posterior cerebral cortex are modulated by the extent of known visuospatial context during recall in a group analysis. We compared neural activation across the visuospatial context conditions (image, panorama, and street) using a vertex-wise ANOVA. We found that three clusters showed significant modulation by known spatial context. By comparing these clusters with our prior work mapping perception and memory-related responses (Steel et al., 2021), we determined that these regions fell anterior to group-average locations of the scene perception areas (white) and within the place memory areas (black). Whole-brain analysis thresholded at FDR-corrected p value (q) < 0.05 (vertex-wise F > 6.4, p < 0.0001). Scene perception areas (white) and place memory areas (black) were defined using data from an independent group of participants (Steel et al., 2021).

Figure 4.

The extent of visuospatial context associated with a scene view modulates place memory area activation during recall. A, For each participant, we compared activation of individually localized scene perception and place memory areas across the visuospatial context conditions during recall using an ANOVA with ROI (perception/memory area), visuospatial context (image [i], panorama [p], street [s]), and hemisphere as factors. Across all cortical surfaces, the place memory area activation was modulated by spatial context to a greater degree than the scene perception areas. Gray dots indicate individual participants. See Figure 5 for control areas. B, Time course of activation during recall (in seconds) shows differential responses of the scene perception and place memory areas. On all cortical surfaces, the place memory areas showed robust activation during recall (4-12 s), while the scene perception areas on the ventral and lateral surfaces did not. Gray lines indicate SEM at each TR (2 s). *p < 0.05. **p < 0.01. ***p < 0.005.

Figure 5.

Analysis of control areas suggests that modulation by visuospatial context during recall is specific to the place memory areas. To determine whether visuospatial context was processed outside of the place memory areas, we considered additional regions for analysis: an early visual area (occipital pole, anatomically defined), a high-level visual area that selectively responds to faces rather than scenes (FFA; functionally defined), and an area known to be involved in mnemonic processing (hippocampus, anatomically defined). No region was modulated by visuospatial context during recall. I, Image; P, panorama; S, street.

Figure 6.

The place memory and scene perception areas represent the identity of the stimulus being recalled. Top, Representational similarity matrix showing stimulus × stimulus correlation of the pattern of activity within each ROI. Each cell represents the similarity of activation patterns between two stimuli derived from an iterative half-split multivoxel pattern analysis. The matrix diagonals represent the average similarity of a stimulus with itself across each data split (identity). Stimuli are grouped by condition. Colored axis labels represent image, panorama, street. Bottom, Identity discrimination index for the place memory and scene perception areas. To ensure that condition-related differences could not explain identity decoding, we examined the ability to discriminate stimulus identity by comparing the average similarity between a stimulus and itself versus other stimuli within its condition. Both the scene perception and place memory areas evidenced significant identity discrimination: *p < 0.05; **p < 0.01; ***p < 0.005.

Figure 7.

The extent of visuospatial context associated with a scene view modulates place memory area activation during the perceptual recognition task. We compared activation of the scene perception and place memory areas across visuospatial context conditions during recognition using an ANOVA with ROI (perception/memory area), visuospatial context (image [i], panorama [p], street [s]), and hemisphere as factors. On the lateral and ventral surfaces, the place memory area activation was modulated by visuospatial context to a greater degree than the scene perception areas. On the medial surface, activity of both the scene perception and place memory area was modulated by known visuospatial context. *p < 0.05. **p < 0.01. ***p < 0.005.

Figure 8.

Analysis of control areas suggests that modulation by visuospatial context during perceptual recognition is specific to the place memory areas. To determine whether visuospatial context was processed outside of the place memory areas, we considered additional regions for analysis: an early visual area (occipital pole, anatomically defined), a high-level visual area that selectively responds to faces rather than scenes (FFA; functionally defined), and an area known to be involved in mnemonic processing (hippocampus, anatomically defined). No regions outside the place memory areas were significantly modulated by visuospatial context during recognition. I, Image; P, panorama; S, street.

Figure 9.

Modulation by spatial context across the brain during recognition. We compared neural activation across the visuospatial context conditions (Image, Panorama, and Street) using a vertex-wise ANOVA. We found that three clusters showed significant modulation by known spatial context, which fell largely within the place memory areas (black). Whole-brain analysis thresholded at FDR-corrected p value (q) < 0.05 (vertex-wise F > 8.1, p < 0.001; q < 0.05). Scene perception areas (white) and place memory areas (black) were defined using data from an independent group of participants (Steel et al., 2021).