A Comprehensive Survey of Depth Completion Approaches

Abstract

Depth maps produced by LiDAR-based approaches are sparse. Even high-end LiDAR sensors produce highly sparse depth maps, which are also noisy around the object boundaries. Depth completion is the task of generating a dense depth map from a sparse depth map. While the earlier approaches focused on directly completing this sparsity from the sparse depth maps, modern techniques use RGB images as a guidance tool to resolve this problem. Whilst many others rely on affinity matrices for depth completion. Based on these approaches, we have divided the literature into two major categories; unguided methods and image-guided methods. The latter is further subdivided into multi-branch and spatial propagation networks. The multi-branch networks further have a sub-category named image-guided filtering. In this paper, for the first time ever we present a comprehensive survey of depth completion methods. We present a novel taxonomy of depth completion approaches, review in detail different state-of-the-art techniques within each category for depth completion of LiDAR data, and provide quantitative results for the approaches on KITTI and NYUv2 depth completion benchmark datasets.

Keywords: depth completion; depth maps; image-guidance.

Figure 9

Qualitative comparison of the top three reported methods on KITTI depth completion test set, including (b) RigNet [33], (c) GuideNet [51], and (d) DySPN [60]. Given sparse depth maps and the input guidance color images (a), the methods output dense depth predictions (1st row). The corresponding error maps (2nd row) are taken from the KITTI leaderboard for comparison. Warmer color represents higher error.

Figure 1

First Column shows the RGB images from two different scenes, the middle column contains the sparse depth maps produced from LiDAR. The last column shows the predicted dense depth maps for the corresponding scenes. (a) RGB Image. (b) LiDAR sparse Depth Map. (c) Prediction.

Figure 2

Approaches to depth completion problem. Unguided approaches utilize either only LiDAR information or confidence maps and LiDAR information for dense depth completion. The image-guided methods (multi-branch and spatial propagation networks) employ guidance images (RGB, semantic maps, surface normals) to guide the process of depth completion. The multi-branch networks can be further divided into guided image filtering methods, which aim to learn useful kernels from one modality and apply it to other modalities.

Figure 3

Early fusion between RGB image and LiDAR sparse depth. At first, both modalities are fused and then sent to the Deep Neural Network for dense depth completion.

Figure 4

Sequential fusion between RGB image and LiDAR sparse depth map. The RGB branch produces color depth, which along with LiDAR sparse depth map, is sent to the depth branch to estimate the final dense depth map.

Figure 5

Late fusion between RGB image and LiDAR sparse depth map. It consists of two separate branches to process RGB images and LiDAR sparse depth maps. Both of the branches produce dense depth maps, which are fused to produce a final dense depth map.

Figure 6

Deep Fusion between RGB image and LiDAR sparse depth map. Each modality is passed from a dedicated branch. The features from the decoder of the RGB branch are fused into the encoder of the depth branch. The symbol “F” represents the fusion operation. Common choices for fusion operation include addition or concatenation. However, complex fusion schemes can also be employed. By the guidance of the RGB branch, the depth branch produces a final dense depth map.

Figure 7

KITTI depth completion benchmark. Part (a) shows the aligned RGB images. Part (b) depicts the sparse LiDAR depth maps, whereas Part (c) represents the dense ground-truth depth maps. Colorization is applied on LiDAR sparse depth maps and corresponding ground-truth to generate visualizations. The highlighted areas are used to show the sparsity in KITTI depth completion benchmark.

Figure 8

Nyu-v2 depth dataset. Part (a) shows the aligned RGB images. Part (b) depicts the sparse Kinect depth maps, which are generated by randomly sampling only 500 points from the ground truth. Part (c) represents the fully dense ground-truth depth maps. Colorization is applied on Kinect sparse depth maps and corresponding ground-truth to generate visualizations. (a) RGB Image. (b) Kinect sparse Depth Map. (c) Ground-truth.