The Treachery of Images: How Realism Influences Brain and Behavior

Abstract

Although the cognitive sciences aim to ultimately understand behavior and brain function in the real world, for historical and practical reasons, the field has relied heavily on artificial stimuli, typically pictures. We review a growing body of evidence that both behavior and brain function differ between image proxies and real, tangible objects. We also propose a new framework for immersive neuroscience to combine two approaches: (i) the traditional build-up approach of gradually combining simplified stimuli, tasks, and processes; and (ii) a newer tear-down approach that begins with reality and compelling simulations such as virtual reality to determine which elements critically affect behavior and brain processing.

Keywords: actions; images; immersive neuroscience; real objects; real-world behavior; virtual reality.

Figure 1:. Methods used to study behavior and brain responses to real objects

Innovative methods used to compare responses to real objects and representations. (A) Example from behavior. (i) In a recent study of decision-making, Romero et al. [57] used a custom-built turntable device to display a large set of real objects and closely-matched 2-D computerized images of everyday snack foods. Schematic shows the experimental setup from above. On real object trials the stimuli were visible on one sector of the turntable; on image trials the stimuli were displayed on a retractable monitor mounted on a sliding platform. Stimulus viewing on all trials was controlled using liquid-crystal glasses that alternated from transparent (‘closed’) to opaque (‘open’) states. (ii) Real object trial (left); image trial (right). Though stimuli are shown from above here, from the participants’ viewpoint, displays appeared similar except for differences in stereopsis. (B) Presenting observers with real objects is especially challenging within fMRI environments. (i) Snow et al. [53] used fMRI to compare brain responses to everyday real-world objects versus photos. Using a repetition-suppression design, pairs of real/picture stimuli were presented from trial-to-trial on a turntable mounted over the participant’s waist. Following from Culham et al., [130], the head coil was tilted forwards to enable participants to view the stimuli directly, without the use of mirrors. (ii) On each trial, two objects (lower left), each mounted on opposite sides of a central partition, were presented in rapid succession. Stimulus viewing on each trial was controlled using time-locked LED illumination; gaze was controlled using a red fixation light (lower right).

Figure 2:. Tearing down vs. building up approaches

Different stimuli can be conceptualized as falling along a continuum of realness, from reduced or artificial (low in ecological validity, as shown in the lower left), to fully real (high in ecological validity, as shown in the upper right). Although ecological validity and control/convenience are thought to trade off, immersive neuroscience approaches can optimize both through well-designed apparatus and protocols. Answering questions about the importance of realness requires a fundamental shift away from the traditional “build-up” approach, in which cognition is studied by making reduced stimuli gradually more complex, to a “tear-down” approach, in which we start by studying responses to fully real stimuli and then gradually remove components. Although “tear-down” and “build-up” approaches may not always yield the same results, combining the two methods will permit a fuller understanding of the cognitive and brain mechanisms that support naturalistic vision and action. For example, the importance of stereopsis as a depth cue may differ between a build-up approach using random-dot stereograms [133] and a tear-down approach in which other depth cues (especially motion parallax) are available. A tear-down approach can reveal whether solids are processed qualitatively differently than artificial stimuli (represented by the distance between vertical dashed gray lines), in which case responses to real-world solids cannot be predicted by those to pictures or virtual stimuli. We postulate that the gap between artificial stimuli vs. real objects may be more quantitative or qualitative depending on the participants’ task and the brain area under study.

Figure I (Box 3).. Which aspects of reality matter?

The flowchart illustrates a sequence to determine and optimize the validity of a proxy for reality.