Vehicle Detection in Aerial Images Based on Region Convolutional Neural Networks and Hard Negative Example Mining

Abstract

Detecting vehicles in aerial imagery plays an important role in a wide range of applications. The current vehicle detection methods are mostly based on sliding-window search and handcrafted or shallow-learning-based features, having limited description capability and heavy computational costs. Recently, due to the powerful feature representations, region convolutional neural networks (CNN) based detection methods have achieved state-of-the-art performance in computer vision, especially Faster R-CNN. However, directly using it for vehicle detection in aerial images has many limitations: (1) region proposal network (RPN) in Faster R-CNN has poor performance for accurately locating small-sized vehicles, due to the relatively coarse feature maps; and (2) the classifier after RPN cannot distinguish vehicles and complex backgrounds well. In this study, an improved detection method based on Faster R-CNN is proposed in order to accomplish the two challenges mentioned above. Firstly, to improve the recall, we employ a hyper region proposal network (HRPN) to extract vehicle-like targets with a combination of hierarchical feature maps. Then, we replace the classifier after RPN by a cascade of boosted classifiers to verify the candidate regions, aiming at reducing false detection by negative example mining. We evaluate our method on the Munich vehicle dataset and the collected vehicle dataset, with improvements in accuracy and robustness compared to existing methods.

Keywords: convolutional neural networks; hard negative example mining; hyper region proposal network; vehicle detection.

Figure 1

Two challenges for vehicle detection in aerial images. (a) small vehicles; (b) hard negative examples.

Figure 2

Proposed vehicle detection framework. Original large scale images are cropped into small scale blocks for training and testing.

Figure 3

Architecture of Hyper Region Proposal Network (HRPN). For $702\times 624$ images, the feature maps size of Conv1-Conv5 are $351\times 312\times 96$, $88\times 79\times 256$, $45\times 40\times 384$, $45\times 40\times 384$, $45\times 40\times 256$, respectively.

Figure 4

Training process of the cascade of boosted classifiers. We adopt the cascade of six stages, and employ {64, 128, 256, 512, 1024, 2048} weak classifiers in each stage.

Figure 5

Ground truth of the original Munich dataset, our modified ground truth and types of vehicles.

Figure 6

Recall versus Intersection-over-Union (IoU) threshold on the Munich Vehicle test set. (a 50 region proposals; (b) 100 region proposals; (c) 150 region proposals.

Figure 7

Recall versus number of proposals for different combining layers (IoU = 0.3).

Figure 8

Performance comparisons of the proposed method, HRPN, H-Faster (HRPN + Fast R-CNN) and aggregated channel features (ACF) detector in terms of mAP values in 10 test images.

Figure 9

precision-recall curve (PRC) of the proposed method and other state-of-the-art approaches for vehicle detection in 10 test images, respectively.

Figure 10

Performance after rescaling the image with different factors.

Figure 11

Performance comparisons of different methods in terms of AP values for the collected vehicle dataset.

Figure 12

Detection results from test images. Red boxes denote correct localization of car, yellow boxes denote correct localization of truck, green boxes and blue boxes denote missing detection and incorrect detection, respectively. (a–d) are results for the Munich test aerial image blocks; (e–h) are results on unmanned aerial vehicle (UAV) images; (i) is results for pansharpened color infrared (CIR) image; (j) is results for the original large-scale Munich test image; (k) is results for a large satellite image of Tokyo.