IMU-Based Gait Recognition Using Convolutional Neural Networks and Multi-Sensor Fusion

Abstract

The wide spread usage of wearable sensors such as in smart watches has provided continuous access to valuable user generated data such as human motion that could be used to identify an individual based on his/her motion patterns such as, gait. Several methods have been suggested to extract various heuristic and high-level features from gait motion data to identify discriminative gait signatures and distinguish the target individual from others. However, the manual and hand crafted feature extraction is error prone and subjective. Furthermore, the motion data collected from inertial sensors have complex structure and the detachment between manual feature extraction module and the predictive learning models might limit the generalization capabilities. In this paper, we propose a novel approach for human gait identification using time-frequency (TF) expansion of human gait cycles in order to capture joint 2 dimensional (2D) spectral and temporal patterns of gait cycles. Then, we design a deep convolutional neural network (DCNN) learning to extract discriminative features from the 2D expanded gait cycles and jointly optimize the identification model and the spectro-temporal features in a discriminative fashion. We collect raw motion data from five inertial sensors placed at the chest, lower-back, right hand wrist, right knee, and right ankle of each human subject synchronously in order to investigate the impact of sensor location on the gait identification performance. We then present two methods for early (input level) and late (decision score level) multi-sensor fusion to improve the gait identification generalization performance. We specifically propose the minimum error score fusion (MESF) method that discriminatively learns the linear fusion weights of individual DCNN scores at the decision level by minimizing the error rate on the training data in an iterative manner. 10 subjects participated in this study and hence, the problem is a 10-class identification task. Based on our experimental results, 91% subject identification accuracy was achieved using the best individual IMU and 2DTF-DCNN. We then investigated our proposed early and late sensor fusion approaches, which improved the gait identification accuracy of the system to 93.36% and 97.06%, respectively.

Keywords: deep convolutional neural network; error minimization; gait identification; inertial motion analysis; multi-sensor fusion; spectro-temporal representation.

Figure 1

Human gait identification task.

Figure 2

The overview of our proposed system for Human Gait Identification.

Figure 3

Resulting Factor Extraction. R indicates resulting factor.

Figure 4

Extracted cycle data sample plot for subject 1 across all the sensors.

Figure 5

Time-frequency representation of one cycle data using different TFDs.

Figure 6

TF representation of gait samples as CNN inputs.

Figure 7

block diagram for early and late score fusion.

Figure 8

The training paradigm for learning the fusion weights.

Figure 9

A data sample of the resulting thresholds on the ${\Theta }_{q,w}(.)$ measure.

Figure 10

CNN architecture.

Figure 11

10 Cross Validation Performance of the Individual Sensors Including Both Gyroscope (Gyro) and Accelerometer (Acc) sensors. $C{V}_j$ indicates Cross-Validation j, and $S_i$ indicates sensor i.

Figure 12

Multi-sensor early fusion performance using min and max pooling.