Restricting Visual Exploration Directly Impedes Neural Activity, Functional Connectivity, and Memory

Abstract

We move our eyes to explore the visual world, extract information, and create memories. The number of gaze fixations-the stops that the eyes make-has been shown to correlate with activity in the hippocampus, a region critical for memory, and with later recognition memory. Here, we combined eyetracking with fMRI to provide direct evidence for the relationships between gaze fixations, neural activity, and memory during scene viewing. Compared to free viewing, fixating a single location reduced: 1) subsequent memory, 2) neural activity along the ventral visual stream into the hippocampus, 3) neural similarity between effects of subsequent memory and visual exploration, and 4) functional connectivity among the hippocampus, parahippocampal place area, and other cortical regions. Gaze fixations were uniquely related to hippocampal activity, even after controlling for neural effects due to subsequent memory. Therefore, this study provides key causal evidence supporting the notion that the oculomotor and memory systems are intrinsically related at both the behavioral and neural level. Individual gaze fixations may provide the basic unit of information on which memory binding processes operate.

Keywords: functional connectivity/similarity; gaze fixations; hippocampus; neuroimaging; visual exploration.

Figure 1

(A). Example stimuli. The scene images vary in clutter (along rows) and space size (along columns). The color-tile images vary in tile size. The clutter and size level were not relevant to the goals of the current study. (B). Scene processing task: Participants were presented with either a green or red fixation cross at the start of each encoding trial. A green fixation cross instructed the participants to freely view the upcoming scene or scrambled color-tile image (free-viewing). A red fixation cross instructed the participants to maintain fixation on the location of the cross during the presentation of the upcoming scene or scrambled color-tile image (fixed-viewing). Participants were instructed to encode as much information regarding the scene or scrambled color-tile image as possible under both viewing conditions. 6 runs of scenes and 2 runs of scrambled color-tile images were presented to the participants, with timing as indicated.

Figure 2

(A). Number of gaze fixations. Participants made more fixations on average per trial under free-viewing versus fixed-viewing instructions. More fixations were elicited by scenes versus color-tile pictures, but only in the free-viewing condition. (B). The distribution (across all participants) of the number of gaze fixations for the scenes (left) and color-tile pictures (right) under fixed (red) and free (blue) viewing conditions. Similar distributions are observed for the number of fixations for the scenes and color-tile pictures. (C). Fixation heatmaps (weighted by fixation durations) for a sample scene in the free-viewing versus fixed-viewing condition illustrates the difference in viewing patterns in the 2 conditions. (D). Average saccade amplitude, which is larger under free-viewing versus fixed-viewing instructions. (E). Recognition memory performance (with confidence considered) was better in the free-viewing versus fixed-viewing condition only for scenes. Recognition memory was calculated by assigning 2 points to stimuli that were correctly recognized with high confidence, 1 point for those correctly recognized with low confidence, 0 points for previously viewed images endorsed as “new” with low confidence, and − 1 point for previously viewed images endorsed as “new” with high confidence. The embedded bar graph shows memory accuracy results calculated using hit rate-false alarm rate. ^* = p < 0.05; ^*** = p < 0.001. x = significant ANOVA interaction effect, p < 0.005, t test with df = 35. Error bars represent Standard Error.

Figure 3

(A). Neural activity difference between the free- and fixed-viewing in the hippocampus (HPC) and parahippocampal place area (PPA) ROIs. Significant responses were observed in the HPC and PPA bilaterally for the free- versus fixed-viewing contrast. This free- versus fixed viewing difference was larger for scenes than for color-tile pictures in the PPA, but not in the HPC at the ROI level. *** = p < 0.001; **** = p < 0.0001. t test with df = 35, one-tailed. Error bars represent Standard Error. (B). Brain section views of the free- versus fixed- viewing activation differences for scenes (left), and color-tile pictures (middle). The scene (free-fixed) versus color-tile pictures (free-fixed) contrast, that is, the interaction effect, is depicted on the right. The zoomed-in image thresholded at p < 0.05 with no corrections shows the voxels within the HPC. The voxel at the crosshair survived Small Volume Correction within the HPC (p_fwe-corr < 0.05). (C). Surface views of the free- versus fixed- viewing activation differences for scenes (left), and color-tile pictures (middle). The scene (free-fixed) versus color-tile pictures (free-fixed) contrast is depicted on the right. The occipital cortex and ventral and medial temporal cortex showed stronger activation during the free-viewing condition, compared to the fixed-viewing condition. Medial temporal lobe regions and the PPA showed a stronger effect of viewing condition for the scenes compared to the color-tile pictures. For illustration purposes, data in B and C are thresholded at p = 0.005, 10-voxel extension with no corrections. For the brain section and surface views, L indicates the left hemisphere and R the right hemisphere.

Figure 4

The number of gaze fixations positively predicts activation in the hippocampus (HPC) and parahippocampal place area (PPA) bilaterally. (A). Results for the anatomical HPC ROIs and functional PPA ROIs (*** = p < 0.001, t test with df = 34, one-tailed. Error bars represent Standard Error.). (B). Activation of a cluster in the bilateral HPC was positively predicted by the number of gaze fixations (p < 0.05 Small volume family-wise error correction). The left HPC: p_fwe-corr = 0.001, p < 0.00001, voxel size = 177 voxels with the peak voxel location = [−22, −30, −2]; for the right HPC: p_fwe-corr = 0.009, p < 0.00001, voxel size = 87 voxels with the peak voxel location = [28, −34, 4]. The zoomed-in image thresholded at p < 0.05 with no corrections shows the voxels within the HPC. The voxel at the crosshair survived Small Volume Correction within the HPC (p_fwe-corr < 0.05). (C). Brain surface views for the prediction of gaze fixations. (D, E), and (F) show the fixation modulation effects for the scenes under fixed viewing, color-tile pictures under free-viewing, and color-tile pictures under fixed-viewing, respectively. For illustration purposes, brain section (B, D, E, F) and surface (C) views are also presented at p < 0.005, 10-voxel extension with no corrections. For the brain section and surface views, L indicates the left hemisphere and R the right hemisphere.

Figure 5

Brain section views of the left (A) and right (C) HPC, for which connectivity with the left (top) and right (bottom) PPA was modulated by viewing condition. Brain surface viewings of free versus fixed viewing modulation effect on functional connectivity for the left (B) and right (D) PPA. For illustration purposes, data were thresholded at p = 0.005, 10-voxel extension with no corrections. The left PPA showed connectivity with a cluster in the left HPC (voxel size = 20 voxels with the peak voxel location = [−22, −22, −10], p_fwe-corr = 0.016, p < 0.001), and the right PPA showed connectivity with a cluster in both the left HPC (voxel size = 58 voxels with the peak voxel location = [−22, −24, −8], p_fwe-corr = 0.001, p < 0.0001) and the right HPC (voxel size = 24 voxels with the peak voxel location = [26, −24, −8], p_fwe-corr = 0.014, p < 0.001) that was stronger in the free- than in the fixed-viewing condition. For the brain section and surface views, L indicates the left hemisphere and R the right hemisphere.

Figure 6

(A). Subsequent memory positively predicts activation in the hippocampus (HPC) and parahippocampal place area (PPA) bilaterally. ROI analysis results were based on anatomical HPC ROIs and functional PPA ROIs. (B). Activation in a cluster in the PPA bilaterally and in the right HPC was positively predicted by the number of gaze fixations and subsequent memory in a conjunction analysis (p = 0.0025, with 50-voxel extension, no corrections). (C). Cross-voxel pattern similarity between the fixation modulation effect and subsequent memory effect in HPC and PPA ROIs are greater in the scene free-viewing than the fixed-viewing condition. Ventral brain surface views of voxel-wise fixation modulation effects (left) and subsequent memory effect (right) showing the ventral and posterior medial temporal lobe regions are each associated with both the number of gaze fixations and subsequent memory performance. Voxel-wise fixation modulation and subsequent memory effect in HPC and PPA at the group level are embedded to illustrate the similarity between the 2 effects in the free-viewing condition. (D). The number of fixations predicted activation in PPA bilaterally after controlling for subsequent memory effect. (E). Activation of a cluster in the PPA and HPC bilaterally was positively predicted by the number of gaze fixations after controlling for subsequent memory effect (p = 0.005, with 10-voxel extension, no corrections). The zoomed-in images thresholded at p < 0.05 with no corrections show the voxels within the HPC. The voxel at the crosshair survived Small Volume Correction within the HPC (p_fwe-corr < 0.05). (F. & G). Brain surface views for the prediction of the number of gaze fixations (F) and subsequent memory (G) for the scenes under free viewing after controlling for shared variability between the 2 predictors. (H). The whole brain cross-voxel pattern similarity between the fixation modulation and subsequent memory is higher in the free- than fixed-viewing conditions for scenes, with or without controlling for the shared variability between the 2 variables. A horizontal section of the whole brain voxel-wise fixation modulation and subsequent memory effect at the group level are embedded to illustrate the similarity between the 2 effects in the free-viewing condition. Note: Surface and section view (B, E, F), and (G) are presented at p < 0.005, 10-voxel extension with no corrections. For the brain section and surface views, L indicates the left hemisphere and R the right hemisphere. For (A, C, D), and (H), + = p < 0.10; * = p < 0.05; ** = p < 0.005; *** = p < 0.001, t test with df = 34, one-tailed. Error bars represent Standard Error.