Depth from a Motion Algorithm and a Hardware Architecture for Smart Cameras

Abstract

Applications such as autonomous navigation, robot vision, and autonomous flying require depth map information of a scene. Depth can be estimated by using a single moving camera (depth from motion). However, the traditional depth from motion algorithms have low processing speeds and high hardware requirements that limit the embedded capabilities. In this work, we propose a hardware architecture for depth from motion that consists of a flow/depth transformation and a new optical flow algorithm. Our optical flow formulation consists in an extension of the stereo matching problem. A pixel-parallel/window-parallel approach where a correlation function based on the sum of absolute difference (SAD) computes the optical flow is proposed. Further, in order to improve the SAD, the curl of the intensity gradient as a preprocessing step is proposed. Experimental results demonstrated that it is possible to reach higher accuracy (90% of accuracy) compared with previous Field Programmable Gate Array (FPGA)-based optical flow algorithms. For the depth estimation, our algorithm delivers dense maps with motion and depth information on all image pixels, with a processing speed up to 128 times faster than that of previous work, making it possible to achieve high performance in the context of embedded applications.

Keywords: FPGA (Field Programmable Gate Array); depth estimation; monocular systems; optical flow; smart cameras.

Figure 1

Block diagram of the proposed algorithm.

Figure 2

The optical flow step: first, curl images $(\overline{{\mathbf{Curl}}_t}(x,y))$, $(\overline{{\mathbf{Curl}}_{t+1}}(x,y))$ are computed. Then, given the curl images for two consecutive frames, pixels displacements ${\Delta }_x(x,y)$, ${\Delta }_y(x,y)$ (optical flow for all pixels in the reference image) are computed using a dynamic template based on the optical flow previously computed $({\Delta }_{x,t-1}(x,y),{\Delta }_{y,t-1}(x,y))$.

Figure 3

Curl computation example. Input image taken from the KITTI benchmark dataset [35].

Figure 4

Optical flow example. Image codification as proposed in the Tsukuba benchmark dataset [36].

Figure 5

The proposed optical flow algorithm formulation: patch size = 10, search size = 10, and sampling value = 2. For each pixel in the reference image $f_t$, n overlapped regions are constructed in $f_{t+1}$, and the n region center that minimizes or maximizes any similarity metric is the tracked position (flow) of the pixel $(x,y)$ at $f_{t+1}$.

Figure 6

(a) Epipolar geometry: depth in the scene is proportional to the disparity value, i.e., far objects have low disparity values, while closer objects are associated with high disparity values. To compute the disparity map (disparities for all pixels in the image) a stereo pair (two images with epipolar geometry) are needed. (b) Single moving camera: in this work we suppose that depth in the scene is proportional to the pixel velocity across the time. To compute the pixel velocity, optical flow across two consecutive frames has to be computed.

Figure 7

Depth estimation using the proposed algorithm.

Figure 8

FPGA architecture for the proposed algorithm.

Figure 9

FPGA architecture for the “frame buffer” unit. Two external memories configured in switching mode makes it possible to store the current frame (time t) into a DRAM configured in write mode, while another DRAM (in read mode) deliver pixel flow for a previous frame (frame at time $t-1$).

Figure 10

FPGA architecture for the optical flow estimation.

Figure 11

The circular buffers architecture. For an $n\times n$ patch, a shift mechanism “control” unit manages the read/write addresses of $n+1$ BRAMs. In this formulation, n BRAMs are in read mode, and one BRAM is in write mode in each clock cycle. The $n\times n$ buffer then delivers logic registers with all pixels within the patch in parallel.

Figure 12

FPGA architecture for the “curl” unit.

Figure 13

FPGA architecture for the “depth estimation” unit.

Figure 14

Accuracy performance for different FPGA-based optical flow algorithms.

Figure 15

Optical flow: quantitative/qualitative results for the KITTI dataset.

Figure 16

Depth estimation: quantitative/qualitative results for the KITTI dataset.

Figure 17

The KITTI dataset: Sequence 00; 3D reconstruction by the proposed approach. Our algorithm provides rough depth maps (a lower accuracy compared with previous algorithms) but with real-time processing and with the capability to be implemented in embedded hardware; as a result, real-time dense 3D reconstructions can be obtained, and these can be exploited by several real world applications such as augmented reality, robot vision and surveillance, and autonomous flying.