Real-Time Abnormal Object Detection for Video Surveillance in Smart Cities

Abstract

With the adaptation of video surveillance in many areas for object detection, monitoring abnormal behavior in several cameras requires constant human tracking for a single camera operative, which is a tedious task. In multiview cameras, accurately detecting different types of guns and knives and classifying them from other video surveillance objects in real-time scenarios is difficult. Most detecting cameras are resource-constrained devices with limited computational capacities. To mitigate this problem, we proposed a resource-constrained lightweight subclass detection method based on a convolutional neural network to classify, locate, and detect different types of guns and knives effectively and efficiently in a real-time environment. In this paper, the detection classifier is a multiclass subclass detection convolutional neural network used to classify object frames into different sub-classes such as abnormal and normal. The achieved mean average precision by the best state-of-the-art framework to detect either a handgun or a knife is 84.21% or 90.20% on a single camera view. After extensive experiments, the best precision obtained by the proposed method for detecting different types of guns and knives was 97.50% on the ImageNet dataset and IMFDB, 90.50% on the open-image dataset, 93% on the Olmos dataset, and 90.7% precision on the multiview cameras. This resource-constrained device has shown a satisfactory result, with a precision score of 85.5% for detection in a multiview camera.

Keywords: camera network; computer vision; deep convolutional network; gun and knife detection; object detection; smart city; video surveillance.

Figure 1

System overview: 1. We first obtain the desired input video or images. 2. On that sequence of frames, the MSD-CNN model is used first. 2.1 Frame classification, 2.2 Frame Localization, and then 2.3 Frame Detection. 3. We can obtain the result.

Figure 2

Taxonomy of algorithm for detecting guns and knives.

Figure 3

Illustration of multiclass subclass sequential flow. The dashed arrow indicates the abnormal subclass; the dotted arrow indicates the normal subclass. The right-side box indicates the major multiclasses; the left-side box indicates their respective subclasses.

Figure 4

The architecture of the MSD-CNN model. Black, orange, green, blue, and red rectangles indicate the convolutional layer (Conv), activation layer (Act), batch normalization layer (BN), max pooling, and dropout, respectively. Light gray filled rectangles indicate the flattened layer; the dark gray rectangle indicates the dense layer. ⨁ indicates the addition of input features.

Figure 5

The sequence of the flow of MSD-CNN methodology.

Figure 6

The accuracy of each subclass with respect to the trained batches of images. Colors representing each aspect are shown in the legend.

Figure 7

Data augmentation technique for dataset.

Figure 8

Two separate experimental plots show the accuracy in order to analyze the training.

Figure 8

Two separate experimental plots show the accuracy in order to analyze the training.

Figure 9

Experimental Training losses of multiclass subclass output classification plotted using matplotlib. They are plotted differently for analysis.

Figure 9

Experimental Training losses of multiclass subclass output classification plotted using matplotlib. They are plotted differently for analysis.

Figure 10

The accuracy of each subclass with respect to the trained batches of images. The red rectangle box determines an abnormal frame, and the blue rectangle box defines the detected subclass.

Figure 10

The accuracy of each subclass with respect to the trained batches of images. The red rectangle box determines an abnormal frame, and the blue rectangle box defines the detected subclass.

Figure 11

Depiction of a false negative, where there are three automatic guns in the background.