Event-Based Motion Capture System for Online Multi-Quadrotor Localization and Tracking

Abstract

Motion capture systems are crucial in developing multi-quadrotor systems due to their ability to provide fast and accurate ground truth measurements for tracking and control. This paper presents the implementation details and experimental validation of a relatively low-cost motion-capture system for multi-quadrotor motion planning using an event camera. The real-time, multi-quadrotor detection and tracking tasks are performed using a deep learning network You-Only-Look-Once (YOLOv5) and a k-dimensional (k-d) tree, respectively. An optimization-based decentralized motion planning algorithm is implemented to demonstrate the effectiveness of this motion capture system. Extensive experimental evaluations were performed to (1) compare the performance of four deep-learning algorithms for high-speed multi-quadrotor detection on event-based data, (2) study precision, recall, and F1 scores as functions of lighting conditions and camera motion, and (3) investigate the scalability of this system as a function of the number of quadrotors flying in the arena. Comparative analysis of the deep learning algorithms on a consumer-grade GPU demonstrates a 4.8× to 12× sampling/inference rate advantage that YOLOv5 provides over representative one- and two-stage detectors and a 1.14× advantage over YOLOv4. In terms of precision and recall, YOLOv5 performed 15% to 18% and 27% to 41% better than representative state-of-the-art deep learning networks. Graceful detection and tracking performance degradation was observed in the face of progressively darker ambient light conditions. Despite severe camera motion, YOLOv5 precision and recall values of 94% and 98% were achieved, respectively. Finally, experiments involving up to six indoor quadrotors demonstrated the scalability of this approach. This paper also presents the first open-source event camera dataset in the literature, featuring over 10,000 fully annotated images of multiple quadrotors operating in indoor and outdoor environments.

Keywords: YOLO; datasets for robotic vision; event-based cameras; k-d tree; motion capture systems; motion coordination; motion planning; multi-quadrotor systems; neural network; object detection; pose estimation.

Figure 1

The event camera was mounted on the ceiling facing downward. Depicted here are 3 quadrotors positioned within the indoor arena, depicted in RGB (left) and event-based (right) camera streams.

Figure 2

Modular multi-process system architecture developed as part of this study featured detection and tracking, and motion planning software services operated as processes on the host machine. The quadrotors received commands from their respective decentralized optimization-based motion planners over WiFi connections.

Figure 3

Various $t_a$ were assessed by training experimental networks before selecting a value for dataset capture. (Left to Right) $t_a$ = 0.05 s, $t_a$ = 0.1 s, and $t_a$ = 0.15 s. The $t_a$ that provided the values of precision and recall closest to 1 were used for all subsequent data and experiments.

Figure 4

Path planning experiment scenarios with multiple quadrotors viewed from above. (a) Square path, (b) circle path, (c) lawnmower path, and (d) cubic spline path. A flight safety corridor (as indicated by solid lines) was defined to enclosed all waypoints. Each quadrotor was commanded by the motion planner to stay within this predefined flight corridor as it traversed the waypoints.

Figure 5

Three categories of experimental evaluations are presented in Section 4. These categories focus on CNN algorithm comparisons, robustness of the motion capture system to environmental conditions, and actual performance evaluation for multiple quadrotors motion planning.

Figure 6

(a) Precision, (b) recall, (c) F1 score evaluated on the YOLOv5 validation set.

Figure 7

Performance metrics for a variety of object detection networks. Two YOLO architectures as well as the Faster R-CNN and RetinaNet methods are compared in terms of precision and recall.

Figure 8

Sampling/inference rates in Hz are shown for two YOLO architectures, Faster R-CNN, and RetinaNet object detection networks.

Figure 9

Detection performance was evaluated under various lighting conditions. LED light strips were used to modulate the brightness of the experiment environment. Also shown here is the Lux meter used for ambient light measurements.

Figure 10

The motion capture system was used to test collision avoidance using a decentralized optimization-based motion planner. In each flight test, the quadrotors were able to successfully avoid $d_{safe}$ violations. Flight tests involved 2 quadrotors (top panel of images) and 3 quadrotors (bottom panel of images) flying toward each other. All quadrotors were operating between the speeds of 0.2 m/s and 1 m/s as determined by their respective motion planners.

Figure 11

Indoor flight tests involving up to six quadrotors in the arena are depicted in RGB (top) and event (bottom) formats.

Figure 12

Experiments involving 2 quadrotors carried out in outdoor environments are depicted in RGB (left) and event (right) format. For the outdoor experiments, the RGB and event cameras were mounted at different locations on the imaging quadrotor. Due to the different camera perspectives, the quadrotors appear to be at slightly different positions in the RGB and event images shown here.