A role for visual areas in physics simulations

Abstract

To engage with the world, we must regularly make predictions about the outcomes of physical scenes. How do we make these predictions? Recent computational evidence points to simulation-the idea that we can introspectively manipulate rich, mental models of the world-as one explanation for how such predictions are accomplished. However, questions about the potential neural mechanisms of simulation remain. We hypothesized that the process of simulating physical events would evoke imagery-like representations in visual areas of those same events. Using functional magnetic resonance imaging, we find that when participants are asked to predict the likely trajectory of a falling ball, motion-sensitive brain regions are activated. We demonstrate that this activity, which occurs even though no motion is being sensed, resembles activity patterns that arise while participants perceive the ball's motion. This finding thus suggests that mental simulations recreate sensory depictions of how a physical scene is likely to unfold.

Keywords: Simulation; fMRI; imagery; intuitive physics.

Figure 1:. Task Design.

(A) An example of a board that constituted the primary stimulus in the ball fall task. Participants had to determine which of the two catchers the ball would land in if dropped (B) A schematic depicting the blocked design of the task variants (Simulation, Perception, Control, and Native), as well as the internal composition of a block. (C) A schematic depicting the trial outlines for each of the three variants of interest. The Native variant was not included in the subsequent fMRI analyses and is hence not shown here.

Figure 2:. Board Designations for Behavioral Analyses.

(A) An example of a board on which the ball only hit two planks. Such boards were classified as having a short simulation length. (B) An example of a board on which the ball hit four planks. Such boards were classified as having a long simulation length. (C) An example of a board where slightly jittering the position of each plank (four jittered examples are shown to the right) had a minimal impact on the ball’s final position. Such boards were classified as having a low simulation uncertainty. (D) An example of a board where slightly jittering the position of each plank (four jittered examples are shown to the right) greatly impacted the ball’s final position. Such boards were classified as having a high simulation uncertainty.

Figure 3:. fMRI Analysis Pipeline.

(A) A schematic of the motion localizer task. The display alternated between blocks of moving and static dots, flanked by periods of fixation. (B) A hypothetical activation map of motion-sensitive voxels derived from a Motion > Static localizer contrast, plotted as a 3D point cloud to demonstrate ROI selection. (C) An example participant’s t-values in the ROI from (B) for each of the three task conditions, contrasted to baseline. Based on these t-maps, we assessed whether the voxel-wise representation in the Simulation condition was more similar to the Perception condition (S-P) or the Control condition (S-C). (D) A line graph showing an example comparison of S-P and S-C similarities. The participant shown in (C) is highlighted in color in (D), and the grey lines represent a hypothetical group effect if the analysis were repeated for all 12 participants.

Figure 4:. Behavioral Results.

(A) Participants’ mean task accuracy across the three variants of interest. Participants were extremely good at all three variants. (B) Participants’ mean reaction times as a function of simulation length and uncertainty. We found that simulation length and simulation uncertainty affected participants’ reaction times on the task. (C) Participants’ average accuracy on the task as a function of simulation length and uncertainty. We found that simulation uncertainty affected participants’ reaction times, whereas simulation length did not. Both of these behavioral effects were previously reported in Ahuja & Sheinberg, 2019. Error bars in all figures represent the standard error of the mean performance for the twelve subjects.

Figure 5:. Localizer Results.

Activation maps for a second level Motion > Static contrast at a p < 0.05 threshold (family-wise error [FWE] cluster corrected for multiple comparisons, extent threshold 187). We observed several canonically motion sensitive regions such as area MT and PPC in this contrast. These voxels were used to define ROIS for subsequent RSA analyses.

Figure 6:. RSA Results.

(A) Pairwise comparisons of S-P and S-C representational similarities in a motion-sensitive ROI. Each pair of points represents one participant. We found that for all participants, the representational similarity between the Simulation and Perception conditions was greater (evidenced by a higher Spearman correlation) than the representational similarity between the Simulation and Control conditions (B) The same analysis as in (A), repeated for an MT ROI. (C) The same analysis as in (A), repeated for a PPC ROI.

Figure 7:. Searchlight Results.

Clusters of voxels that were highlighted by a searchlight analysis for consistently exhibiting the main effect from Figure 5. The searchlight largely revealed the same regions as we had previously isolated using the motion localizer task (slices here are the same as in Figure 4).