Deep generative models with data augmentation to learn robust representations of movement intention for powered leg prostheses

Abstract

Intent recognition is a data-driven alternative to expert-crafted rules for triggering transitions between pre-programmed activity modes of a powered leg prosthesis. Movement-related signals from prosthesis sensors detected prior to movement completion are used to predict the upcoming activity. Usually, training data comprised of labeled examples of each activity are necessary; however, the process of collecting a sufficiently large and rich training dataset from an amputee population is tedious. In addition, covariate shift can have detrimental effects on a controller's prediction accuracy if the classifier's learned representation of movement intention is not robust enough. Our objective was to develop and evaluate techniques to learn robust representations of movement intention using data augmentation and deep neural networks. In an offline analysis of data collected from four amputee subjects across three days each, we demonstrate that our approach produced realistic synthetic sensor data that helped reduce error rates when training and testing on different days and different users. Our novel approach introduces an effective and generalizable strategy for augmenting wearable robotics sensor data, challenging a pre-existing notion that rehabilitation robotics can only derive limited benefit from state-of-the-art deep learning techniques typically requiring more vast amounts of data.

Keywords: data augmentation; deep learning; intent recognition; prosthesis control; signal processing.

Fig. 1.. Training data collection.

(Left) Collecting ramp descent training data from a subject in the ambulation laboratory with a circuit-based protocol. (Right) Collecting stair descent training data from another subject in a therapy gym which more closely resembles community ambulation.

Fig. 2.. Global data augmentation using transformations based on prior knowledge.

(Top left) Jitter by additive zero-centered Gaussian noise (σ = 0.1) for training the denoising autoencoder. (Top right) Two shifted copies at ± 10 ms relative to the original window account for variation in event detection timing. (Bottom left) Eight scaled copies multiplied by a uniformly sampled scaling factor account for baseline shift. (Bottom right) Ten combined shifted-scaled copies.

Fig. 3.. Proposed deep generative model, a semi-supervised denoising adversarial autoencoder.

(A) Individual modules with output dimensions listed in parentheses above each layer. N is the number of convolution kernels, K is the kernel size, S is the stride length, LReLU represents the leaky ReLU activation, and tanh represents the hyperbolic tangent activation. The latent space dimensionality (zdim) was set to 10. “Real” samples for the latent space and label discriminators were sampled from a multivariate standard normal distribution and a categorical distribution (Cat), respectively. (B) Overall schematic of network connectivity.

Fig. 4.. Overall offline error rates (mean ± standard error) for individual and pooled user configurations.

(A) Individual user configuration. (B) Pooled user configuration. Data from one or two experimental sessions were globally augmented (GA) (left panels). Globally augmented one-day basline data were combined with synthetic examples from specific augmentation by reconstruction (Recon) and/or by sampling (Gen) using the trained autoencoder (right panels). Other generative model-based strategies included manifold alignment (Aligned) and classification using the encoder (CNN). Dashed lines represent the globally augmented one-day and two-day baselines and the Day 3 leave-one-out (LOO) cross-validation benchmark.

Fig. 5.. Representative signals (normalized) using different generative model-based strategies.

(Top to bottom) Knee velocity from TF3 (individual user) in level-ground mode for HCLW, ankle position from TF2 (individual user) in ramp descent mode for HCRD, load from TF1 (pooled users) in stair descent mode for HCSD, G_z for TF4 (pooled users) in stair ascent for MSW, A_y for TF2 (individual user) for level-ground mode for HCRD, knee reference torque for TF1 (individual user) for level-ground mode for MSW. In columns 1–4, colored traces represent samples from the corresponding training data. In column 5, colored traces represent the corresponding test data. Black traces in columns 2–4 represent artificial sensor data generated using specific augmentation by reconstruction (Recon), specific augmentation by sampling (Gen), or manifold alignment (Align). The dashed pink lines represent the mean of the training data for columns 1–3 and the mean of the testing data for columns 4–5.