Internal feedback in the cortical perception-action loop enables fast and accurate behavior

Abstract

Animals move smoothly and reliably in unpredictable environments. Models of sensorimotor control, drawing on control theory, have assumed that sensory information from the environment leads to actions, which then act back on the environment, creating a single, unidirectional perception-action loop. However, the sensorimotor loop contains internal delays in sensory and motor pathways, which can lead to unstable control. We show here that these delays can be compensated by internal feedback signals that flow backward, from motor toward sensory areas. This internal feedback is ubiquitous in neural sensorimotor systems, and we show how internal feedback compensates internal delays. This is accomplished by filtering out self-generated and other predictable changes so that unpredicted, actionable information can be rapidly transmitted toward action by the fastest components, effectively compressing the sensory input to more efficiently use feedforward pathways: Tracts of fast, giant neurons necessarily convey less accurate signals than tracts with many smaller neurons, but they are crucial for fast and accurate behavior. We use a mathematically tractable control model to show that internal feedback has an indispensable role in achieving state estimation, localization of function (how different parts of the cortex control different parts of the body), and attention, all of which are crucial for effective sensorimotor control. This control model can explain anatomical, physiological, and behavioral observations, including motor signals in the visual cortex, heterogeneous kinetics of sensory receptors, and the presence of giant cells in the cortex of humans as well as internal feedback patterns and unexplained heterogeneity in neural systems.

Keywords: internal feedback; optimal control; sensorimotor control; speed–accuracy trade-off.

Fig. 1.

Single-loop model of sensorimotor control. The organism receives information from the external environment via sensors, communicates this information through the body, computes actions, and then acts on the environment; this forms the external feedback loop, or single loop model (black). Internal signals that flow opposite to the direction of the external feedback loop are classified as internal feedback (pink). Thus, the internal feedback is counterdirectional. Internal feedback also includes lateral interactions within an area or between areas at the same processing stage (not shown).

Fig. 2.

A partial, simplified schematic of sensorimotor control. We focus on key cortical and subcortical areas and communications between them. Black and green arrows indicate communications that traverse from sensing toward actuation; green arrows are particularly fast pathways, which enable the tracking of moving objects in our model. Pink arrows indicate internal feedback signals, which traverse from actuation toward sensing. Solid lines indicate direct neuronal projections, while broken lines include both direct and indirect connections. SC, spinal cord; Th, thalamus; V1, primary visual cortex; M1+, primary motor cortex and additional motor areas; V2/3, secondary and tertiary visual cortex; IT, inferotemporal cortex; MT, mediotemporal cortex (V5). Only a subset of the internal feedback pathways are shown (e.g., not included are internal feedback signals from M1+ to V2 and IT).

Fig. 3.

Optimal control model for a system with sensor delays. Tracking error $x$ is sensed, and then communicated by the sensor with some delay to the $K_1$ block, which computes the appropriate actuation. Counterdirectional internal feedback (pink) conveys information from actuation back toward sensing. Internal computation $K_2$ adjusts the sensor signal to compensate for actions taken by the system; this results in improved performance.

Fig. 4.

Internal feedback improves performance when there are internal delays in sensing. The scalar problem of tracking a moving target over a line was simulated, varying the task difficulty ($\alpha$ = spectral radius of $A$, representing the dynamics). The “Ideal” controller contains no sensor delays. The “Internal Feedback” controller contains sensor delays, and uses internal feedback to compensate for the delays. The “No Internal Feedback” controller contains sensor delays but uses no internal feedback. As $\alpha$ approaches 2, the task becomes infeasible without internal feedback (broken line). Shaded areas indicate SDs.

Fig. 5.

Internal feedback in a controller with instantaneous but imperfect sensing and actuation. $A$, $B$, and $C$ represent the state, actuation, and sensing matrices of the physical plant; $K$ represents the optimal controller, and $L$ represents the optimal observer. The Time Shift block shifts $\widehat{x}(t+1)$ to $\widehat{x}(t)$ in Eq. 5. The internal feedback pathways (blue) are inherent to the Kalman filter; these use state, actuation, and sensing models to create an internal estimate of the tracking error or state. All internal feedback depicted in this diagram is counterdirectional and assumed to have no delay and infinite bandwidth.

Fig. 6.

Internal feedback in a controller with sensor and actuator delays. $A$, $B$, and $C$ represent the state, actuation, and sensing matrices of the physical plant; $K_1,K_2,L_1,and{L}_2$ are submatrices of the optimal controller and observer gains. The internal feedback pathways (pink) through $L_2$ and $K_2$ compensate for sensor and actuator delays, respectively. Other internal feedback pathways (blue) are inherent to the Kalman filter. All internal feedback depicted in this diagram is counterdirectional. The yellow box contains parts of the controller that roughly correspond to motor areas in the cortex.

Fig. 7.

Optimal localized control of two coupled subsystems. (Top) Overall schematic. Each subsystem has its own corresponding local controller, which senses and actuates only its assigned subsystem. Local controllers communicate to each other via lateral internal feedback (pink), with some delay. (Bottom) Circuitry of local controller 1. Local controller 2 has identical circuitry, with different matrices; $A_{12}$ instead of $A_{21}$, $A_{22}$ instead of $A_{11}$, etc.

Fig. 8.

Localization of function within the motor-related cortex: Although different parts of the cortex control different parts of the body, these parts of the body are inherently mechanically coupled. As a result, internal feedback is useful and in some cases necessary to maintain localization of function. In simulations, we consider the problem of tracking a moving target over a two-dimensional space, varying the task difficulty. The “Ideal” controller is centralized (i.e., no delays between local controllers) and obtains the best performance. The localized controller with internal feedback achieves similar performance. The localized controller without internal feedback suffers from substantially worse performance (higher cost). As task difficulty increases, the task becomes infeasible without internal feedback (broken line). Shaded areas indicate SDs.

Fig. 9.

Optimal control model of attention, with moveable sensor. (Top) Model with one communication path, in which information is quantized by quantizer $Q$ and conveyed to the controller with delay $T$. (Bottom) Model with two communication paths, and two separate quantizers $Q_s$, $Q_f$, and respective delays. This model can be considered lateral (e.g., V1-V1) or counterdirectional (V2-V1) internal feedback (pink) between the two controller paths.

Fig. 10.

(Left) Internal feedback and layering achieve superior performance when sensor-controller communications are subject to speed–accuracy trade-offs. The “No Internal Feedback” controller uses one layer, while the “Internal Feedback” controller uses two layers, with internal feedback between the layers. The two-layer case consists of a fast-forward pathway compensated by slow internal feedback, which takes slow background changes into account; this achieves better performance (Lower cost) than the case without internal feedback. The “Ideal” controller, where the sensor directly senses the moving object, is also shown. The layered system with internal feedback achieves performance close to the ideal. $T$ represents delay. For the “No Internal Feedback” controller, it represents the delay of the single layer; for the “Internal Feedback” controller, it represents the delay of the slow layer, i.e., $T=T_s$. The delay of the fast layer is held constant. (Right) Performance (log cost) of the two-layer controller with internal feedback as delays of both layers are varied. Performance is best when $T_f$ is low and $T_s$ is sufficiently high.