Perceptual-Cognitive Integration for Goal-Directed Action in Naturalistic Environments

Abstract

Real-world actions require one to simultaneously perceive, think, and act on the surrounding world, requiring the integration of (bottom-up) sensory information and (top-down) cognitive and motor signals. Studying these processes involves the intellectual challenge of cutting across traditional neuroscience silos, and the technical challenge of recording data in uncontrolled natural environments. However, recent advances in techniques, such as neuroimaging, virtual reality, and motion tracking, allow one to address these issues in naturalistic environments for both healthy participants and clinical populations. In this review, we survey six topics in which naturalistic approaches have advanced both our fundamental understanding of brain function and how neurologic deficits influence goal-directed, coordinated action in naturalistic environments. The first part conveys fundamental neuroscience mechanisms related to visuospatial coding for action, adaptive eye-hand coordination, and visuomotor integration for manual interception. The second part discusses applications of such knowledge to neurologic deficits, specifically, steering in the presence of cortical blindness, impact of stroke on visual-proprioceptive integration, and impact of visual search and working memory deficits. This translational approach-extending knowledge from lab to rehab-provides new insights into the complex interplay between perceptual, motor, and cognitive control in naturalistic tasks that are relevant for both basic and clinical research.

Figure 1.

Measuring coordinated action with different experimental tools. A, Top, The neural control of eye-hand and body coordination can be probed using MRI-compatible tablets and eye-trackers in the scanner or by developing portable setups (e.g., portable EEG system). Middle, Using a robotic manipulandum allows experimental control of the visual and movement space (e.g., mechanical perturbations) and a head-fixed setup enables well-calibrated high-precision eye tracking, while investigating real object manipulation. Bottom, Studies using virtual reality setups or head-mounted eye-tracking glasses allow the study of eye-hand coordination in naturalistic environments. B, Schematic represents how different experimental tools vary along two axes. The ability by the experimenter to control the visual and physical environment (x axis) and the translation of the observed behavior to the real world (y axis). Green boxes represent behavioral methods. Gray boxes represent neuroimaging techniques.

Figure 2.

Schematic overview of the major brain regions and pathways involved in goal-directed eye-hand coordination. Pathways for vision, object and motion perception, eye movements, and top-down cognitive strategies are integrated with cortical, subcortical, and cerebellar networks for sensorimotor transformations to produce coordinated action. A, Left brain lateral view. B, Right brain medial view. LOC, Lateral occipital cortex; MT, middle temporal area; PPC, posterior parietal cortex; SPL 7, superior parietal lobule area 7; SPL 5, superior parietal lobule area 5; IPL, inferior parietal lobule; S1, somatosensory cortex; M1, primary motor cortex; PMC, premotor cortex, SMA, supplementary motor area; PMd, dorsal premotor cortex; FEF, frontal eye fields; PMv, ventral premotor cortex; dlPFC, dorsolateral PFC; CBM, cerebellum; SC, superior colliculus; LGN, lateral geniculate nucleus; PPA, parahippocampal place area; LG, lingual gyrus; Cn, cuneus. Created with Biorender.com.

Figure 3.

Visual signals for action. A, Visual scenes consist of “grammar” that is defined by building blocks in a hierarchical structure, consisting of phrases (top), global objects (middle), and associated local objects (bottom) (Võ, 2021). B, When reaching to stationary objects, the target object is commonly fixated throughout the reach, which allows the integration of foveal vision of the object and peripheral vision of the hand. C, When intercepting moving objects, the eyes either track the moving object with SPEMs or fixate on the expected interception location to guide the hand toward the object.

Figure 4.

The role of optic flow on visually guided steering. A, Drivers are immersed in a simulated environment seen through a head-mounted display with integrated eye tracking. B, Exemplary view inside the virtual reality as participants attempt to keep their head centered within a parameterized and procedurally generated roadway. This image is superimposed with a computational estimate of optic flow indicated here as white arrows (Matthis et al., 2018). C, To assess the effect of cortical blindness on the visual perception of heading will require the development of computational models of visually guided steering that account for the blind field. For illustrative purposes, we have superimposed the results from a Humphrey visual field test on the participant's view at a hypothetical gaze location with approximate scaling (reprinted from Cavanaugh et al., 2015 with permission).

Figure 5.

Movement impairment in stroke survivors. A, Kinesthetic Matching Task where a robotic manipulandum (Kinarm Exoskeleton Lab) passively moved the more affected arm and stroke survivors mirror-matched with the less affected arm (left). Middle, Performance from a stroke survivor that performed well with vision of the limb (top, left and right) and without vision of the limb (top, left); other participants performed worse than control participants with and without the use of vision (bottom, left and right), although vision significantly improved performance for some (right, top and bottom). Distribution of participant performance demonstrates that only 12% of stroke survivors (total N = 261) used vision to effectively correct proprioceptively referenced movements, suggesting that vision often fails to effectively compensate for multisensory impairments (right) (Semrau et al., 2018). B, Left, Saccades made by a healthy control (top) and stroke survivor (bottom) during hand movement (green) and hand dwell on target (yellow) in the Trails Making task. Excessive saccades in stroke survivors are associated with deficits in working memory and top-down visual search (middle). When stroke survivors make saccades during reaching, those movements tend to be slower compared with when they do not make saccades during reaching (right panels).