Human-monkey gaze correlations reveal convergent and divergent patterns of movie viewing

Abstract

The neuroanatomical organization of the visual system is largely similar across primate species, predicting similar visual behaviors and perceptions. Although responses to trial-by-trial presentation of static images suggest that primates share visual orienting strategies, these reduced stimuli fail to capture key elements of the naturalistic, dynamic visual world in which we evolved. Here, we compared the gaze behavior of humans and macaques when they viewed three different 3-minute movie clips. We found significant intersubject and interspecies gaze correlations, suggesting that both species attend a common set of events in each scene. Comparing human and monkey gaze behavior with a computational saliency model revealed that interspecies gaze correlations were driven by biologically relevant social stimuli overlooked by low-level saliency models. Additionally, humans, but not monkeys, tended to gaze toward the targets of viewed individual's actions or gaze. Together, these data suggest that human and monkey gaze behavior comprises converging and diverging informational strategies, driven by both scene content and context; they are not fully described by simple low-level visual models.

Figure 1. Video scenes with superimposed gaze traces

Example frames from the Life of Mammals (A), the Jungle Book (B), and City Lights (C), shown with superimposed gaze coordinates (monkeys green, humans blue).

Figure 2. Correlation in gaze location across scanpaths

Correlations are evident across humans, monkeys, and between species. Here, scanpaths are split into vertical and horizontal coordinates, and plotted for (A) two humans paths viewing City Lights (local correlation r=0.66), (B) a human and a monkey viewing the Jungle Book (local correlation r=0.08), and (C) one monkey during repeated viewings of the Life of Mammals (local correlation r=0.36). The local correlations are typical of the inter-human, inter-species, and intra-monkey scanpaths, respectively. Significant correlations existed between primate scanpaths produced in response to the same video clip. Both spatial position (D) and eye movement speed (E) were correlated across primate scanpaths, whether produced by the same individual or a different individual and whether produced by a human (blue) or a monkey (green). Gray bars indicate the permutation baseline for α = 0.05 (thick) or 0.001 (thin): all primate gaze ISCs were significant with α < .001. Artificial scanpaths produced by a low-level visual saliency model (orange) were significantly correlated with primate scanpaths in spatial position (α < 0.001) and gaze shift timing (α < 0.05); however, residual inter-primate correlations (yellow) were essentially unchanged despite partialling out shared similarities to artificial scanpaths. Finally, to confirm that behavioral correlations were driven by visual fixation priorities, we compared the percentage of gaze samples which overlapped (±3.5°) across different scanpaths (F): The pattern of results was identical to the pattern observed for inter-subject correlation. Furthermore, we found that samples that overlapped between humans and monkeys rarely overlapped with the best-performing artificial scanpath (2 of 31%, see inset), These data rule out the hypothesis that gaze correlations were driven primarily by low-level visual features, at least as characterized by well-established neuromorphic computational saliency models [27, 29].

Figure 3. Comparison of gaze reliability across time, by species

The standard deviation of simultaneously-recorded gaze coordinates from humans (A), monkeys (B) and both species (C) can, in conjunction, differentiate the visual strategies of monkeys or humans. At right we show frames during a clip from the Life of Mammals that most dispersed attention (D, a long shot featuring a number of monkeys), that sometimes gathered attention (E, social stimuli captured sustained human interest, while newly-visible scenery during a pan was quickly surveyed by monkeys). Faces often captured the attention of all primates (F), while dyadic social interactions sometimes produced separate gaze clusters for humans and macaques (G).

Figure 4. Higher-level social cues contribute to observed gaze correlations

We selected video frames in which observed gaze was highly clustered at a location far from that chosen by the low-level saliency model, examined scene content at the human, monkey, and simulated gaze coordinates, and tallied the percentage of frames in which scene content comprised biologically-relevant stimuli. Relative to image regions selected by artificial scanpaths, regions that were consistently viewed by humans and monkeys often featured social agents and the targets of their actions or attentions. Humans and monkeys fixated bodies and faces significantly more than low-level simulations, especially the eye and (for humans) mouth regions. Humans, but not monkeys, fixated socially-cued regions – for example, things being reached or gazed toward – significantly more often than predicted by the model. This effect was not mediated just by hand motion, which attracted gaze significantly more often for the simulation than for actual humans or monkeys.