Improved STNNet, A benchmark for detection, tracking, and counting crowds using Drones

Abstract

In computer vision, navigating multi-object tracking in crowded scenes poses a fundamental challenge with broad applications ranging from surveillance systems to autonomous vehicles. Traditional tracking methods encounter difficulties associating noisy object detections and maintaining consistent labels across frames, particularly in scenarios like video surveillance for crowd control and public safety. This paper introduces 'Improved Space-Time Neighbor-Aware Network (STNNet),' an advanced framework for online Multi-Object Tracking (MOT) designed to address these challenges. Expanding upon the foundational STNNet architecture, our enhanced model incorporates deep reinforcement learning techniques to refine decision-making. By framing the online MOT problem as a Markov Decision Process (MDP), Improved STNNet learns a sophisticated policy for data association, adeptly handling complexities such as object birth/death and appearance/disappearance as state transitions within the MDP. Through extensive experimentation on benchmark datasets, including the MOT Challenge, our proposed Improved STNNet demonstrates superior performance, surpassing existing methods in demanding, crowded scenarios. This study showcases the effectiveness of our approach and lays the groundwork for advancing real-time video analysis applications, particularly in dynamic, crowded environments. Additionally, we utilize the dataset provided by STNNET for density map estimation, forming the basis for our research.•Develop an advanced framework for online Multi-Object Tracking (MOT) to address crowded scene challenges, particularly improving object association and label consistency across frames.•Explore integrating Deep Reinforcement learning techniques into the MOT framework, framing the problem as an MDP to refine decision-making and handle complexities such as object birth or death and appearance or disappearance transitions.

Keywords: Crowd counting; Density estimation; Improved STNNet; Neural network; Surveillance; Tracking and localization.

Graphical abstract

Fig. 1

(a) the localization subnet. $\left(\widehat{X}i,\widehat{y}i\right)$ represent the estimated coordinates of the i th target. I represent the input image.

Fig. 2

(a) the association subnet using [9](b) the neighboring context loss. Notably, the dashed modules in (a) are only used in the training phase. For clarity, we only display the calculation of the terms from time t − 1 to time t in the neighboring context loss [9]. $\left(\widehat{X}i,\widehat{y}i\right)$ represent the estimated coordinates of the i th target. I represent the input image.

Fig. 3

Methodology.