Recurrent Network Solutions for Human Posture Recognition Based on Kinect Skeletal Data

Abstract

Ambient Assisted Living (AAL) systems are designed to provide unobtrusive and user-friendly support in daily life and can be used for monitoring frail people based on various types of sensors, including wearables and cameras. Although cameras can be perceived as intrusive in terms of privacy, low-cost RGB-D devices (i.e., Kinect V2) that extract skeletal data can partially overcome these limits. In addition, deep learning-based algorithms, such as Recurrent Neural Networks (RNNs), can be trained on skeletal tracking data to automatically identify different human postures in the AAL domain. In this study, we investigate the performance of two RNN models (2BLSTM and 3BGRU) in identifying daily living postures and potentially dangerous situations in a home monitoring system, based on 3D skeletal data acquired with Kinect V2. We tested the RNN models with two different feature sets: one consisting of eight human-crafted kinematic features selected by a genetic algorithm, and another consisting of 52 ego-centric 3D coordinates of each considered skeleton joint, plus the subject's distance from the Kinect V2. To improve the generalization ability of the 3BGRU model, we also applied a data augmentation method to balance the training dataset. With this last solution we reached an accuracy of 88%, the best we achieved so far.

Keywords: ambient assisted living; deep learning; human action recognition; recurrent neural network; skeletal data.

Figure 1

Example of the four postures acquired with the Kinect V2 camera. Standing posture (top left panel), sitting posture (top right panel), lying posture (bottom left panel) and dangerous–sitting posture consisting in slumping on a chair with the head leaned backward (bottom right panel) are depicted in the Kinect V2 spatial reference system. In the visualization, the red lines indicate the body segments of the head, trunk, and pelvis. The green lines represent the body segments of the right hemi-body, including the shoulder, arm, forearm, hand, thigh, leg, and foot. Similarly, the blue lines represent the body segments of the left hemi-body.

Figure 2

The 17 joints skeleton considered from each Kinect V2 recording [30] Each number corresponds to a specific point of repere of the body used to reconstruct the skeletal stick diagram.

Figure 3

Numerosity of the sequences before (blue bars) and after (yellow bars) the data augmentation process adopted to increase the cardinality of Class 3 and 4.

Figure 4

The architecture of the 3BGRU model.

Figure 5

Mean confusion matrix obtained over 30 3BGRU architecture simulations with the first dataset.

Figure 6

Mean confusion matrix obtained by 30 3BGRU architecture simulations with the second dataset.

Figure 7

Mean confusion matrix obtained over 30 3BGRU architecture simulations with the third dataset.

Figure 8

Mean classification error for each class. The red bars represent the percentage of FN for each class, while the blue bars represent the percentage of FN that corresponds to frames that were previously considered transitions and now are labeled as pertaining to the class.

Figure 9

Mean confusion matrix obtained by 30 2BLSTM architecture simulations with the third dataset.

Figure 10

Mean classification error for each class for the 3BGRU model on the third dataset. The red bars represent the percentage of FN for each class, while the blue bars represent the percentage of FN that corresponds to frames that were previously considered transitions and now are labeled as pertaining to the class.

Figure 11

Mean confusion matrix obtained from the 30 3BGRU network simulations on the fourth dataset.

Figure 12

Mean classification error for each class. The red bars represent the percentage of FN for each class, while the blue bars represent the percentage of FN that corresponds to frames that were previously considered transitions and now are labeled as pertaining to the class.

Figure 13

Average ROC curves of each class considered. For each graph, the curve referred to the data of the first dataset 3BGRU (blue line) is superimposed on that of the second dataset 3BGRU (red line). The confidence intervals of each average ROC curve are not shown, for purposes of clarity.

Figure 14

Sensitivity (A) and specificity (B) mean results + SD.

Figure 15

The average ROC curves over the 30 simulations for each class, respectively, for third dataset 3BGRU, third dataset 2BLSTM and fourth dataset 3BGRU. The confidence intervals of each ROC curve are not shown, for purposes of clarity.

Figure 16

Mean sensitivity results + SD (A) and mean specificity results + SD (B).