A Systematic Evaluation of Feature Encoding Techniques for Gait Analysis Using Multimodal Sensory Data

Abstract

This paper addresses the problem of feature encoding for gait analysis using multimodal time series sensory data. In recent years, the dramatic increase in the use of numerous sensors, e.g., inertial measurement unit (IMU), in our daily wearable devices has gained the interest of the research community to collect kinematic and kinetic data to analyze the gait. The most crucial step for gait analysis is to find the set of appropriate features from continuous time series data to accurately represent human locomotion. This paper presents a systematic assessment of numerous feature extraction techniques. In particular, three different feature encoding techniques are presented to encode multimodal time series sensory data. In the first technique, we utilized eighteen different handcrafted features which are extracted directly from the raw sensory data. The second technique follows the Bag-of-Visual-Words model; the raw sensory data are encoded using a pre-computed codebook and a locality-constrained linear encoding (LLC)-based feature encoding technique. We evaluated two different machine learning algorithms to assess the effectiveness of the proposed features in the encoding of raw sensory data. In the third feature encoding technique, we proposed two end-to-end deep learning models to automatically extract the features from raw sensory data. A thorough experimental evaluation is conducted on four large sensory datasets and their outcomes are compared. A comparison of the recognition results with current state-of-the-art methods demonstrates the computational efficiency and high efficacy of the proposed feature encoding method. The robustness of the proposed feature encoding technique is also evaluated to recognize human daily activities. Additionally, this paper also presents a new dataset consisting of the gait patterns of 42 individuals, gathered using IMU sensors.

Keywords: classification; feature encoding; gait analysis; human activity recognition; time series sensory data.

Figure 1

A set of sensing modalities that can be used for gait data collection.

Figure 2

Classification of existing feature encoding techniques using sensory data based on their underlying computing methods.

Figure 3

An illustration of codebook-based feature encoding technique.

Figure 4

An illustration of the Random Forest classifier.

Figure 5

The proposed deep CNN consists of three convolutional, two sub-sampling, one flattened, and one dense layer.

Figure 6

An illustration of the proposed LSTM-based deep network to classify the multimodal sensory data.

Figure 7

An illustration of IMU placement in collected dataset to capture the motion information of different body parts.

Figure 8

A summary of the recognition results of four datasets using all the encoding methods. The terms HC, CB, DL, SVM, RF, CNN, and LSTM represent handcrafted features, codebook features, deep learning features, Support Vector Machine, Random Forest, convolutional neural network, and long short-term memory, respectively.