Embedded Vision Intelligence for the Safety of Smart Cities

Abstract

Advances in Artificial intelligence (AI) and embedded systems have resulted on a recent increase in use of image processing applications for smart cities' safety. This enables a cost-adequate scale of automated video surveillance, increasing the data available and releasing human intervention. At the same time, although deep learning is a very intensive task in terms of computing resources, hardware and software improvements have emerged, allowing embedded systems to implement sophisticated machine learning algorithms at the edge. Additionally, new lightweight open-source middleware for constrained resource devices, such as EdgeX Foundry, have appeared to facilitate the collection and processing of data at sensor level, with communication capabilities to exchange data with a cloud enterprise application. The objective of this work is to show and describe the development of two Edge Smart Camera Systems for safety of Smart cities within S4AllCities H2020 project. Hence, the work presents hardware and software modules developed within the project, including a custom hardware platform specifically developed for the deployment of deep learning models based on the I.MX8 Plus from NXP, which considerably reduces processing and inference times; a custom Video Analytics Edge Computing (VAEC) system deployed on a commercial NVIDIA Jetson TX2 platform, which provides high level results on person detection processes; and an edge computing framework for the management of those two edge devices, namely Distributed Edge Computing framework, DECIoT. To verify the utility and functionality of the systems, extended experiments were performed. The results highlight their potential to provide enhanced situational awareness and demonstrate the suitability for edge machine vision applications for safety in smart cities.

Keywords: EdgeX Foundry; artificial intelligence; deep learning; edge; embedded machine vision; smart cities.

Figure 1

DECIoT architecture. The four main different layers of the proposed edge computing framework. From left to right: (i) the Device Service Layer acts as an interface of the system with physical devices and is tasked with the functionality of collecting data and actuating the devices with command; (ii) the Core Services Layer is used for storing data, commanding and registering devices; (iii) the Support Services Layer includes microservices for local/edge analytics and typical application duties, such as logging, scheduling, and data filtering; (iv) the Application Services Layer consists of one or more microservices to extract, transform, and send data from the previous layer to other endpoints or applications.

Figure 2

S4AllCities schema with ICCS (VAEC via NVIDIA Jetson RTX2) and Tekniker (I.MX8M Plus) edge platforms transmitting video streaming and the number of people detected.

Figure 3

S4AllCities Hardware platform based on the I.MX8M PLUS and enclosed as a security inspection camera.

Figure 4

NXP Image Signal Processing Software (optimized by OpenCL 1.2 and OpenVX 1.1) [75].

Figure 5

eIQ block diagram presenting the tools available to develop, analyze and deploy a custom model bringing your own data or bringing your own model [76].

Figure 6

Inference Engines and libraries available for Neuronal Network Model Deployment for the NXP I.MX8M Plus platforms [77].

Figure 7

Setup scenario and person detection results obtained with the I.MX8M Plus-based embedded board. Left, non-overlapped persons detected. Right, overlapped persons detected.

Figure 8

Details of the total dataset. Top left indicates that the dataset has only one class, i.e., Person. Top right shows the shapes of the bounding boxes, as well as their orientation. Bottom left depicts the location of the center of each bounding box in the total dataset images in which darker pixels imply that more bounding boxes in these areas exist. Bottom right shows the width and height of the bounding boxes.

Figure 9

Basic training metrics of the custom model.

Figure 10

Training set metrics.

Figure 11

Validation set metrics.

Figure 12

Observed FN entries (yellow ellipse with dashed line) and FP entries (magenta indications) during the person detection process applying the pre-trained YOLOv5s.

Figure 13

Observed FN entries (yellow ellipse with dashed line) and FP entries (white indications) during the person detection process applying the custom YOLOv5s.