Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Feb 11;22(4):1381.
doi: 10.3390/s22041381.

Contextual Detection of Pedestrians and Vehicles in Orthophotography by Fusion of Deep Learning Algorithms

Masoomeh Shireen Ansarnia  1 , Etienne Tisserand  1 , Patrick Schweitzer  1 , Mohamed Amine Zidane  1 , Yves Berviller  1
Affiliations

Contextual Detection of Pedestrians and Vehicles in Orthophotography by Fusion of Deep Learning Algorithms

Masoomeh Shireen Ansarnia et al. Sensors (Basel). .

Abstract

In the context of smart cities, monitoring pedestrian and vehicle movements is essential to recognize abnormal events and prevent accidents. The proposed method in this work focuses on analyzing video streams captured from a vertically installed camera, and performing contextual road user detection. The final detection is based on the fusion of the outputs of three different convolutional neural networks. We are simultaneously interested in detecting road users, their motion, and their location respecting the static environment. We use YOLOv4 for object detection, FC-HarDNet for background semantic segmentation, and FlowNet 2.0 for motion detection. FC-HarDNet and YOLOv4 were retrained with our orthophotographs dataset. The last step involves a data fusion module. The presented results show that the method allows one to detect road users, identify the surfaces on which they move, quantify their apparent velocity, and estimate their actual velocity.

Keywords: FC-HarDNet; FlowNet 2.0; YOLO; deep learning; optical flow; orthophotography; semantic segmentation.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
The concept of contextual detection in our project.
Figure 2
Figure 2
Illustration of the main objectives of the project.
Figure 3
Figure 3
Diurnal (a) and nocturnal (b) test bench.
Figure 4
Figure 4
Segmentation of validation images.
Figure 5
Figure 5
Segmentation of unseen orthophotographs, obtained with a drone at an altitude of 10 m.
Figure 6
Figure 6
Typical cases of poor segmentation of road users with FC-HarDNet.
Figure 7
Figure 7
Pedestrian detection by YOLOv4 under the illumination of 8 lux. Exposure parameters: aperture = 3.5, shutter speed = 1/30, and ISO = 400 (a), 800 (b), and 1600 (c).
Figure 8
Figure 8
Inaccuracy of pedestrian detection in the presence of a shadow.
Figure 9
Figure 9
Examples of optical flow estimation with FlowNet 2.0 (middle) and PWC-Net (bottom), tested on our video sequences.
Figure 10
Figure 10
The overall structure of the fusion process.
Figure 11
Figure 11
Block diagram of the fusion system.
Figure 12
Figure 12
Location of the bottom of the detected objects.
Figure 13
Figure 13
Detection, segmentation, and analysis of the movement of a cyclist. (Top): YOLOv4 detection. (Center): fusion of the three algorithms. (Bottom): quantified contextual information (type, positioning, and speed of the user, the nature of the surface).
Figure 14
Figure 14
Detection, segmentation, and analysis of the movement of a car and a pedestrian. (Top): YOLOv4 detection. (Center): fusion of the three algorithms. (Bottom): quantified contextual information (type, positioning, and speed of the user, the nature of the surface).
See this image and copyright information in PMC

References

    1. Barthélemy J., Verstaevel N., Forehead H., Perez P. Edge-Computing Video Analytics for Real-Time Traffic Monitoring in a Smart City. Sensors. 2019;19:2048. doi: 10.3390/s19092048. - DOI - PMC - PubMed
    1. Chen L.-C., Sheu R.-K., Peng W.-Y., Wu J.-H., Tseng C.-H. Video-Based Parking Occupancy Detection for Smart Control System. Appl. Sci. 2020;10:1079. doi: 10.3390/app10031079. - DOI
    1. Rezaei M., Azarmi M. DeepSOCIAL: Social Distancing Monitoring and Infection Risk Assessment in COVID-19 Pandemic. Appl. Sci. 2020;10:7514. doi: 10.3390/app10217514. - DOI
    1. Silva P.B., Andrade M., Ferreira S. Machine Learning Applied to Road Safety Modeling: A Systematic Literature Review. J. Traffic Transp. Eng. 2020;7:775–790. doi: 10.1016/j.jtte.2020.07.004. - DOI
    1. Tran D., Tan Y.K. Sensorless Illumination Control of a Networked LED-Lighting System Using Feedforward Neural Network. IEEE Trans. Ind. Electron. 2014;61:2113–2121. doi: 10.1109/TIE.2013.2266084. - DOI

LinkOut - more resources