Generative model-enhanced human motion prediction

Abstract

The task of predicting human motion is complicated by the natural heterogeneity and compositionality of actions, necessitating robustness to distributional shifts as far as out-of-distribution (OoD). Here, we formulate a new OoD benchmark based on the Human3.6M and Carnegie Mellon University (CMU) motion capture datasets, and introduce a hybrid framework for hardening discriminative architectures to OoD failure by augmenting them with a generative model. When applied to current state-of-the-art discriminative models, we show that the proposed approach improves OoD robustness without sacrificing in-distribution performance, and can theoretically facilitate model interpretability. We suggest human motion predictors ought to be constructed with OoD challenges in mind, and provide an extensible general framework for hardening diverse discriminative architectures to extreme distributional shift. The code is available at: https://github.com/bouracha/OoDMotion.

Keywords: deep learning; generative models; human motion prediction; variational autoencoders.

FIGURE 1

Graph convolutional network (GCN) network architecture with variational autoencoder (VAE) branch. Here, $n_z=16$ is the number of latent variables per joint

FIGURE A1

(A) Distribution of short‐term training instances for actions in H3.6M. (B) Distribution of training instances for actions in CMU

FIGURE A2

Confusion matrix for a multiclass classifier for action labels. In each case, we use the same input convention ${\overrightarrow{x}}_k=\left[x_{k,1},\dots ,x_{k,N},x_{k,N+1},\dots ,x_{k,N+T}\right]$ _, where $x_{k,n}=x_{k,N}$ for $n\ge N$. Such that in each case input to the classifier is $48\times 20=960$. The classifier has four fully connected layers. Layer 1: input dimensions × 1024, layer 2: $1024\times 512$, layer 3: $512\times 128$, layer 4: $128\times 15$ (or $128\times 8$ for CMU). Where the final layer uses a softmax to predict the class label. Cross entropy loss is used for training and ReLU activations with a dropout probability of 0.5. We used a batch size of 2048, and a learning rate of 0.00001.H3.6M dataset. $N=10$, $T=10$. Number of discrete cosine transformations (DCT) coefficients = 20 (lossesless transformation)

FIGURE A3

Confusion matrix for a multiclass classifier for action labels. In each case, we use the same input convention ${\overrightarrow{x}}_k=\left[x_{k,1},\dots ,x_{k,N},x_{k,N+1},\dots ,x_{k,N+T}\right]$ _, where $x_{k,n}=x_{k,N}$ for $n\ge N$. Such that in each case input to the classifier is $48\times 20=960$. The classifier has four fully connected layers. Layer 1: input dimensions × 1024, layer 2: $1024\times 512$, layer 3: $512\times 128$, layer 4: $128\times 15$ (or $128\times 8$ for CMU). Where the final layer uses a softmax to predict the class label. Cross entropy loss is used for training and ReLU activations with a dropout probability of 0.5. We used a batch size of 2048, and a learning rate of 0.00001.H3.6M dataset. $N=50$, $T=10$. Number of discrete cosine transformations (DCT) coefficients = 20, where the 40 highest frequency DCT coefficients are culled

FIGURE A4

Confusion matrix for a multiclass classifier for action labels. In each case, we use the same input convention ${\overrightarrow{x}}_k=\left[x_{k,1},\dots ,x_{k,N},x_{k,N+1},\dots ,x_{k,N+T}\right]$ _, where $x_{k,n}=x_{k,N}$ for $n\ge N$. Such that in each case input to the classifier is $48\times 20=960$. The classifier has four fully connected layers. Layer 1: input dimensions × 1024, layer 2: $1024\times 512$, layer 3: $512\times 128$, layer 4: $128\times 15$ (or $128\times 8$ for CMU). Where the final layer uses a softmax to predict the class label. Cross entropy loss is used for training and ReLU activations with a dropout probability of 0.5. We used a batch size of 2048, and a learning rate of 0.00001. CMU dataset. $N=10$, $T=25$. Number of discrete cosine transformations (DCT) coefficients = 35 (losses less transformation)

FIGURE A5

Latent embedding of the trained model on both the H3.6M and the CMU datasets independently projected in 2D using UMAP from 384 dimensions for H3.6M, and 512 dimensions for CMU using default hyperparameters for UMAP. (A) H3.6M. All actions, opacity=0.1. (B) H3.6M. All actions in blue: opacity=0.1. Walking in red: opacity=1. (C) CMU. All actions in blue: opacity = 0.1

FIGURE A6

Network architecture with discriminative and variational autoencoder (VAE) branch

FIGURE A7

Graph convolutional layer (GCL) and a residual graph convolutional block (GCB)