Feature Representation and Data Augmentation for Human Activity Classification Based on Wearable IMU Sensor Data Using a Deep LSTM Neural Network

Abstract

Wearable inertial measurement unit (IMU) sensors are powerful enablers for acquisition of motion data. Specifically, in human activity recognition (HAR), IMU sensor data collected from human motion are categorically combined to formulate datasets that can be used for learning human activities. However, successful learning of human activities from motion data involves the design and use of proper feature representations of IMU sensor data and suitable classifiers. Furthermore, the scarcity of labelled data is an impeding factor in the process of understanding the performance capabilities of data-driven learning models. To tackle these challenges, two primary contributions are in this article: first; by using raw IMU sensor data, a spectrogram-based feature extraction approach is proposed. Second, an ensemble of data augmentations in feature space is proposed to take care of the data scarcity problem. Performance tests were conducted on a deep long term short term memory (LSTM) neural network architecture to explore the influence of feature representations and the augmentations on activity recognition accuracy. The proposed feature extraction approach combined with the data augmentation ensemble produces state-of-the-art accuracy results in HAR. A performance evaluation of each augmentation approach is performed to show the influence on classification accuracy. Finally, in addition to using our own dataset, the proposed data augmentation technique is evaluated against the University of California, Irvine (UCI) public online HAR dataset and yields state-of-the-art accuracy results at various learning rates.

Keywords: data augmentation; deep learning; feature representation; human activity recognition; inertial measurement unit sensor; long short term memory.

Figure 1

Human activity recognition system workflow.

Figure 2

Data collection architecture.

Figure 3

An example of 3D raw data for (a) sitting and (b) walking based on 1IMU sensor tied on the left-hand wrist.

Figure 4

(a) An example of time domain data and their spectrogram representing walking data; (b) Proposed feature extraction algorithm.

Figure 5

Data augmentation workflow.

Figure 6

Set 1: OR dataset.

Figure 7

Set 1: Generating local averages.

Figure 8

Set 2: OR + LA1 dataset.

Figure 9

Randomly shuffled (OR + LA1 + SH) feature set 2.

Figure 10

Generating local averages of the shuffled feature set.

Figure 11

Set 3: OR + LA1 + SH + LA2 feature set with local averages.

Figure 12

Visualizing the variance of data to check the influence of each data augmentation block. (a,b,c) represent the variance of the unaugmented dataset, augmented dataset after the first local averaging and that of the augmented dataset after the first local averaging, shuffling and the second local averaging procedure respectively for the standing activity. (d,e,(f) represent the variance of the unaugmented dataset, augmented dataset after the first local averaging and that of the augmented dataset after the first local averaging, shuffling and the second local averaging procedure respectively for the sitting activity. (g,h,i) represent the variance of the unaugmented dataset, augmented dataset after the first local averaging and that of the augmented dataset after the first local averaging, shuffling and the second local averaging procedure respectively for the walking activity.

Figure 13

LSTM cell.

Figure 14

(a) Accuracy versus learning rate based on only the OR dataset without augmentation, (b) precision versus learning rate based on only the OR dataset without augmentation, (c) recall versus learning rate based on only the OR dataset without augmentation and (d) f1_score versus learning rate based on only the OR dataset without augmentation.

Figure 15

100-feature vector dataset: (a) Accuracy versus learning rate, (b) precision versus learning rate, (c) recall versus learning rate and (d) f1_score versus learning rate.

Figure 16

200-feature vector dataset: (a) Accuracy versus learning rate, (b) precision versus learning rate, (c) recall versus learning rate and (d) f1_score versus learning rate.

Figure 17

UCI’s 128-feature vector dataset: (a) Accuracy versus learning rate, (b) precision versus learning rate, (c) recall versus learning rate and (d) f1_score versus learning rate.

Figure 17

UCI’s 128-feature vector dataset: (a) Accuracy versus learning rate, (b) precision versus learning rate, (c) recall versus learning rate and (d) f1_score versus learning rate.

Figure 18

Comparing the performance of OR and OR + LA1 using the UCI dataset and our dataset at various learning rates of: (a) 0.0002 (b) 0.003 (c) 0.006 and (d) 0.01.