Active Vision in Binocular Depth Estimation: A Top-Down Perspective

Abstract

Depth estimation is an ill-posed problem; objects of different shapes or dimensions, even if at different distances, may project to the same image on the retina. Our brain uses several cues for depth estimation, including monocular cues such as motion parallax and binocular cues such as diplopia. However, it remains unclear how the computations required for depth estimation are implemented in biologically plausible ways. State-of-the-art approaches to depth estimation based on deep neural networks implicitly describe the brain as a hierarchical feature detector. Instead, in this paper we propose an alternative approach that casts depth estimation as a problem of active inference. We show that depth can be inferred by inverting a hierarchical generative model that simultaneously predicts the eyes' projections from a 2D belief over an object. Model inversion consists of a series of biologically plausible homogeneous transformations based on Predictive Coding principles. Under the plausible assumption of a nonuniform fovea resolution, depth estimation favors an active vision strategy that fixates the object with the eyes, rendering the depth belief more accurate. This strategy is not realized by first fixating on a target and then estimating the depth; instead, it combines the two processes through action-perception cycles, with a similar mechanism of the saccades during object recognition. The proposed approach requires only local (top-down and bottom-up) message passing, which can be implemented in biologically plausible neural circuits.

Keywords: action-perception cycles; active inference; active vision; depth perception; predictive coding.

Figure 1

Information processing in neural networks (left) and Predictive Coding (right). In a neural network, the visual observation ${\mathit{s}}_v$ travels through the cortical hierarchy in a bottom-up way, detecting increasingly more complex features ${\mathit{x}}^{(i,j)}$ and eventually estimating the depth of an object $\mathit{d}$. The descending projections are considered here as feedback signals that convey backpropagation errors. In contrast, in a Predictive Coding Network the depth $\mathit{d}$ is a high-level belief generating a visual prediction that is compared with the observation. This process leads to a cascade of prediction errors ${\mathit{\epsilon }}^{(i,j)}$ associated with each intermediate prediction ${\mathit{x}}^{(i,j)}$ that are minimized throughout the hierarchy, eventually inferring the correct belief (for details, see the Section 2).

Figure 2

(A) An example of a portion of a kinematic plant. (B) Factor graph of a single level j of the hierarchical kinematic model composed of intrinsic ${\mathit{\mu }}_i^{\left(j\right)}$ and extrinsic ${\mathit{\mu }}_e^{\left(j\right)}$ beliefs. These beliefs generate proprioceptive and visual predictions ${\mathit{p}}_p^{\left(j\right)}$ and ${\mathit{p}}_v^{\left(j\right)}$ through generative models ${\mathit{g}}_p$ and ${\mathit{g}}_v$, respectively. Furthermore, the beliefs predict trajectories (here, only the velocities $\mathit{\mu }{{}^{\prime }}_i^{\left(j\right)}$ and $\mathit{\mu }{\prime }_e^{\left(j\right)}$) through the dynamics functions ${\mathit{f}}_i^{\left(j\right)}$ and ${\mathit{f}}_e^{\left(j\right)}$. Note that the extrinsic belief of level $j-1$ acts as a prior for layer j through a kinematic generative model ${\mathit{g}}_e$. See [21] for more details.

Figure 3

Projection of a 3D point in the camera plane (only two dimensions are shown). The y coordinates of the real point $r_y$ and projected point $p_y$ are related by the ratio between the focal length f and the real point depth $r_z$.

Figure 4

Representation of the hierarchical relationships of a generalized model with homogeneous transformations. The belief over a reference frame ${\mathit{\mu }}_r$ of level i is passed to a function ${\mathit{g}}_t$ encoding a homogeneous transformation along with a belief over a particular transform ${\mathit{\mu }}_t$ (e.g., the angle for rotation or length for translation), generating the reference frame of level $i+1$.

Figure 5

Neural-level implementation of a hierarchical generative model to estimate the depth of a point through Active Inference. The small squares indicate inhibitory connections. Unlike a neural network, depth is estimated by first generating two predictions ${\mathit{p}}_r^{\left(i\right)}$ of the point relative to each eye from a point in the absolute coordinates ${\mathit{\mu }}_a$ and vergence-accommodation angles ${\mathit{\mu }}_{\theta }$. This new belief is in turn used to compute a projection ${\mathit{p}}_c^{\left(i\right)}$ and finally a visual prediction ${\mathit{p}}_v^{\left(i\right)}$. The predictions are then compared with the visual observations, generating prediction errors throughout the hierarchy and eventually driving the beliefs at the top toward the correct values. Note that eye movements are directly triggered to suppress the proprioceptive prediction error ${\mathit{\epsilon }}_p$. Intentional eye movements (e.g., for target fixation) can instead be achieved by setting a prior in the dynamics function ${\mathit{f}}_c$ of the belief over the projected point ${\mathit{\mu }}_c$ (note that for better readability the figure only shows the dynamics function ${\mathit{f}}_c$.

Figure 6

Sequence of time frames of a depth estimation task with simultaneous target fixation. The agent uses alternating action–perception phases to avoid becoming stuck during the minimization process. Each frame is composed of three images: a third-view perspective of the overall task (top) and a first-view perspective consisting of the projection of the target to the respective camera planes of each eye (bottom left and bottom right). In the top panel, the eyes are represented by blue circles and the real and estimated target positions are shown in red and orange. The fixation trajectory (when vergence occurs) is represented in cyan. The thin blue lines are the fixation angles of the eyes. In the bottom panel, the real and estimated target positions are shown in red and orange. The abscissa and ordinate respectively represent the target depth and its projection.

Figure 7

Simulation results. Performance of the depth estimation task with nonuniform (top) and uniform (bottom) foveal resolution during inference with the eyes parallel and fixed (infer parallel), inference with the eyes fixating on the target (infer vergence), and simultaneous inference and target fixation (active vision). The accuracy (left panel) measures the number of trials in which the agent successfully predicts the 2D position of the target, the mean error (middle panel) measures the distance between the real and estimated target positions at the end of every trial, and the time (right panel) measures the number of steps needed to correctly estimate the target.