Deep Learning for Real-Time 3D Multi-Object Detection, Localisation, and Tracking: Application to Smart Mobility

Abstract

In core computer vision tasks, we have witnessed significant advances in object detection, localisation and tracking. However, there are currently no methods to detect, localize and track objects in road environments, and taking into account real-time constraints. In this paper, our objective is to develop a deep learning multi object detection and tracking technique applied to road smart mobility. Firstly, we propose an effective detector-based on YOLOv3 which we adapt to our context. Subsequently, to localize successfully the detected objects, we put forward an adaptive method aiming to extract 3D information, i.e., depth maps. To do so, a comparative study is carried out taking into account two approaches: Monodepth2 for monocular vision and MADNEt for stereoscopic vision. These approaches are then evaluated over datasets containing depth information in order to discern the best solution that performs better in real-time conditions. Object tracking is necessary in order to mitigate the risks of collisions. Unlike traditional tracking approaches which require target initialization beforehand, our approach consists of using information from object detection and distance estimation to initialize targets and to track them later. Expressly, we propose here to improve SORT approach for 3D object tracking. We introduce an extended Kalman filter to better estimate the position of objects. Extensive experiments carried out on KITTI dataset prove that our proposal outperforms state-of-the-art approches.

Keywords: 3D multi-object; deep learning; distance estimation; localisation; object detection; smart mobility; tracking.

Figure 1

Overview of the proposed system composed of three main components: object detection, depth estimation and localisation and tracking.

Figure 2

Mask-RCNN results (returns the mask of each detected object with their class and their confidence score).

Figure 3

Result of YOLOv3 [2] (Returns the position, class and score of trust of each detected object).

Figure 4

Performance comparison between YOLOv3 (left) and SSD (right) detectors.

Figure 5

Results of disparity maps obtained using monocular approaches: (a) original image; (b) MonoResMatch; (c) SfmLearner; (d) Monodepth and (e) Monodepth2.

Figure 6

Results of disparity maps obtained using stereoscopic approaches: (a) original image; (b) stereo-baseline approach; (c) stereo-WLS filter and (d) MADNet.

Figure 7

RMSE error over a sequence of arround 900 frames from the KITTI dataset. The blue, orange, green and red curves correspond respectively to the results of monodepth, monodepth2, MonoResMatch and sfmLearner approaches.

Figure 8

RMSE error over a sequence of around 900 frames from the KITTI dataset. The blue, orange and green curves correspond respectively to the results of Stereo-WLS Filter, Stereo-baseline and MADNet approaches.

Figure 9

Object detection and localisation results over a sample from the KITTI dataset.

Figure 10

Result of the tracking approach over an indoor scene using the stereo images provided by the Intel RealSense D435 sensor. The left frame is acquired at instant $t=1$ and the right one at the instant $t+3$. At $t=1$, we assign an ID to each detected object. Then, at $t+3$, the tracklet is validated and we display the estimated speed.

Figure 11

Tracking results in the road environment.

Figure 12

Tracking results over a sequence from the KITTI dataset. On top, four RGB frames coming from a road sequence acquired at different times with the corresponding tacking boxes of the moving objects and their speed values. On bottom, the maps shown are proposed by this work in order to generate a synthetic tool that allows simple yet comprehensive results overview of motions and distances of the tracked objects.