Human detection from a mobile robot using fusion of laser and vision information

Abstract

This paper presents a human detection system that can be employed on board a mobile platform for use in autonomous surveillance of large outdoor infrastructures. The prediction is based on the fusion of two detection modules, one for the laser and another for the vision data. In the laser module, a novel feature set that better encapsulates variations due to noise, distance and human pose is proposed. This enhances the generalization of the system, while at the same time, increasing the outdoor performance in comparison with current methods. The vision module uses the combination of the histogram of oriented gradients descriptor and the linear support vector machine classifier. Current approaches use a fixed-size projection to define regions of interest on the image data using the range information from the laser range finder. When applied to small size unmanned ground vehicles, these techniques suffer from misalignment, due to platform vibrations and terrain irregularities. This is effectively addressed in this work by using a novel adaptive projection technique, which is based on a probabilistic formulation of the classifier performance. Finally, a probability calibration step is introduced in order to optimally fuse the information from both modules. Experiments in real world environments demonstrate the robustness of the proposed method.

Figure 1.

Schematic diagram of the proposed method.

Figure 2.

Projection of laser segmented clusters to the image plane using the calibration parameters. In the left image, the shaded area indicates the field of view of the camera. The color and number of each cluster are the same in both views.

Figure 3.. Histogram of oriented gradients descriptor

(a) The histogram of oriented gradients (HoG) descriptor is computed over image cells (in red) and 2 × 2 overlapping blocks of cells (green and cyan); (b) visualization of the HoG descriptor computed for the same image

Figure 4.. Fixed and adaptive projection methods

(a) Projection of the same cluster for different corresponding sizes. The cyan rectangle is used in the fixed case, and the larger green window is used in the proposed adaptive method. (b) In the adaptive method, the region of interest (ROI) is scanned by an overlapping sliding window over multiple scales. In this figure, the maximum scale of the pyramid is depicted and the sliding window is of the size 2.0 × 1.0 meters.

Figure 5.. Calibration of the laser and camera

Two reference frames are shown, one with respect to the camera (x_c, y_c, x_c) and another with respect to the laser (x_l, y_l, x_l). The extrinsic calibration provides the matrices, R, T, containing the rotation and translation between the two frames. I corresponds to the 2D image plane. The laser plane, L, is defined by y_l = 0.

Figure 6.

Platform and sensors used in experiments. (a) Summit XL mobile platform; (b) Hokuyo laser range finder above the Firefly camera.

Figure 7.. Laser feature sets—detection error tradeoff curves

In both figures, the green line corresponds to the basis feature set, the red line to the basis plus the distance features and the blue line to the proposed feature set. (a) The training and test sets are acquired from the split up of same experimental data: solid lines indicate the gymnasium and dotted lines the outdoors data sets, respectively. (b) Solid lines signify that the outdoor data set is used for training and its gymnasium counterpart for testing. Dotted lines signify the reverse procedure.

Figure 8.. Gymnasium experiment detection error tradeoff curves

The curves are plotted for each sensor individually and, also, for the three fusion techniques. Fixed-size (a) and adaptive (b) projection techniques are compared.

Figure 9.. Outdoors experiment detection error tradeoff curves

The curves are plotted for each sensor individually and, also, for the three fusion techniques. Fixed-size (a) and adaptive (b) projection techniques are compared.

Figure 10.

Detector images from the gymnasium (a), (b) and outdoor experiments (c), (d). The boxes above each cluster give the probability of the laser and image detection modules and the mean fused probability. Cluster distance is shown at the bottom of the enclosing rectangle. (a) Typical scenario with multiple people, movement in various directions and occlusions. (b) The laser module does not detect the right person correctly. Due to correct camera classification, the person is eventually detected after fusion. (c) Outdoors, the laser detector performance is considerably lower. (d) The right person (blue rectangle) is misclassified from both sensor modules. The person in the middle is too close to the camera; in this case, the cluster is evaluated by the laser classifier only. Persons hidden by the bench on the right are not detected by the laser.