Deep Learning-Based Object Detection, Localisation and Tracking for Smart Wheelchair Healthcare Mobility

Abstract

This paper deals with the development of an Advanced Driver Assistance System (ADAS) for a smart electric wheelchair in order to improve the autonomy of disabled people. Our use case, built from a formal clinical study, is based on the detection, depth estimation, localization and tracking of objects in wheelchair's indoor environment, namely: door and door handles. The aim of this work is to provide a perception layer to the wheelchair, enabling this way the detection of these keypoints in its immediate surrounding, and constructing of a short lifespan semantic map. Firstly, we present an adaptation of the YOLOv3 object detection algorithm to our use case. Then, we present our depth estimation approach using an Intel RealSense camera. Finally, as a third and last step of our approach, we present our 3D object tracking approach based on the SORT algorithm. In order to validate all the developments, we have carried out different experiments in a controlled indoor environment. Detection, distance estimation and object tracking are experimented using our own dataset, which includes doors and door handles.

Keywords: computer vision; deep learning; distance estimation; distance measurement; object detection; object localization; semantic map; smart mobility; tracking.

Figure 1

MCIndoor2000 dataset image sample (top row), custom dataset (bottom row).

Figure 2

IRSEEM Electric Wheelchair. Note: the $T265$ RealSense camera used for odometry was not mounted for this photo. For the experiments described in the text, it was placed near the $D435$ camera.

Figure 3

ROS Software Architecture.

Figure 4

Example of objects detection in wheelchair indoor environment including doors and handles.

Figure 5

A 3D local semantic map from a hallway. Bottom image: the wheelchair. Top image: objects (doors) in the environment.

Figure 6

Arrangement of doors in the validation dataset. The ground truth bounding boxes are displayed in red with the associated label overlaid in yellow. In the top-right of figure, we can see a part of the Vicon motion capture system within cameras network.

Figure 7

Distribution of IoU values for the detected doors.

Figure 8

ANL environment including: Wheelchair, doors, doors handles, and Vicon cameras ground truth system.

Figure 9

Measured distance between correctly classified objects and $D435$ RealSense camera (orange), ground truth distance (blue), absolute value of the difference between the ground truth distance and the distance estimated by the camera (green). The detected objects are sorted by class, then detection order.

Figure 10

Distribution of the relative distance measurement error.

Figure 11

Distribution of the relative distance measurement error, relative to the distance of the detected objects, sliding 5 points average.