Indoor Group Identification and Localization Using Privacy-Preserving Edge Computing Distributed Camera Network

Abstract

Social interaction behaviors change as a result of both physical and psychiatric problems, and it is important to identify subtle changes in group activity engagements for monitoring the mental health of patients in clinics. This work proposes a system to identify when and where group formations occur in an approximately 1700 m² therapeutic built environment using a distributed edge-computing camera network. The proposed method can localize group formations when provided with noisy positions and orientations of individuals, estimated from sparsely distributed multiview cameras, which run a lightweight multiperson 2-D pose detection model. Our group identification method demonstrated an F1 score of up to 90% with a mean absolute error of 1.25 m for group localization on our benchmark dataset. The dataset consisted of seven subjects walking, sitting, and conversing for 35 min in groups of various sizes ranging from 2 to 7 subjects. The proposed system is low-cost and scalable to any ordinary building to transform the indoor space into a smart environment using edge computing systems. We expect the proposed system to enhance existing therapeutic units for passively monitoring the social behaviors of patients when implementing real-time interventions.

Keywords: Cameras; group position detection; group position estimation; pose estimation.

Fig. 1.

Layout of the indoor space spanning 1700 m² in our study. Our study site has various regions to provide physical and cognitive training for individuals with mild cognitive impairment relating to activities in daily living. These areas include a gym, dining area, kitchen, lounge, activity area, tech bar, and staff zone. The proposed work could successfully identify and localize social groups across these regions.

Fig. 2.

Overall pipeline for group identification and localization. The frames obtained by the cameras are processed in real time on the edge computing camera device to obtain the poses and bounding boxes of individuals in the study space. These are used to estimate the positions and facing orientations of individuals in the space. The false positives in the estimated positions and orientations are removed following which groups are identified and localized using DBSCAN. This figure is borrowed from Kwon et al. [25].

Fig. 3.

Camera installation in the study site. (a) Field of view (RED) of each camera (Sony IMX219 8-megapixel sensor) is 62.2°, and 39 cameras (BLUE) can cover most regions in our study space. Some regions, such as the kitchen and lounge, are covered by more cameras (five cameras) showing darker red shades. This is due to differences in the number of accessible power and network sources in the ceiling to connect edge computing systems. (b) Camera setup on the ceiling. The edge computing system is concealed in the ceiling and linked to the nearest network and power outlets. (c) Google Coral TPU USB Accelerator enables the edge computing systems to run deep learning models in real time (1 Hz) to detect 2-D poses from videos. The figure is adapted from Kwon et al. [25].

Fig. 4.

Distance metric calculation using sectors. (a) Sectors obtained by weighting distances up to 2.1 m and angles away from facing angle up to 80° on either side of the individual. These sectors are used to calculate a distance metric between two individuals based on the overlap of sectors of the two individuals. (b) Sectors of two individuals overlapping when they are facing each other at a distance of 5.5 m. (c) Of two individuals overlapping when they are facing almost parallel at a distance of 5.5 m.

Fig. 5.

Examples of data collection in the study space with their corresponding ground truth. The images in the top and bottom rows are the raw images captured by the camera. The locations of individuals from camera images are shown on the map in the second row using boxes and arrows. The top left image is taken in the activity area. The top right image is taken in the gym area. The bottom two images are taken in the lounge and tech bar. We simulated realistic group activities often observed from real patients with MCI in our study site. In the map, the circles of the same color depict individuals belonging to the same group. The same colored circles are used in the camera images to show the positions of individuals on the map.

Fig. 6.

We removed all detections where the positions and orientations did not change for a certain amount of time S. This figure shows the precision, recall, F1 score, and MAE as a function of S, the preprocessing window, for removing false positive samples. The overall performance increases when S increases from 0 to 6 s, and remains relatively flat beyond this point.

Fig. 7.

Qualitative analysis comparing ground truth (left) with P (middle) and P+O (right). The dots represent the positions of individuals and the lines represent their orientations. Markers of the same color represent members of the same group. The red markers indicate individuals not part of any group. (a) Example in the kitchen area. P (middle), without orientation information, assigns all individuals into one group. P+O identifies both groups correctly using the orientation information. (b) Example in the gym area. P (middle) identifies both groups correctly even though it misses detecting one person in the larger group. P+O (right) does not identify the group with two people because the estimated orientations of the members of this group are facing away from each other. (a) Kitchen. (b) Gym.

Fig. 8.

Examples of room dividers placed in the built environment. The image on the left shows a reflective room divider placed in the gym area and the image on the right shows a nonreflective room divider placed in the activity area. The positions of these room dividers are shown on the map in the middle image. The presence of the room dividers cause occlusions, thus, resulting in some people not being picked up by the cameras, leading to false negative detections of individuals which affects group localization.