PA-Tran: Learning to Estimate 3D Hand Pose with Partial Annotation

Abstract

This paper tackles a novel and challenging problem-3D hand pose estimation (HPE) from a single RGB image using partial annotation. Most HPE methods ignore the fact that the keypoints could be partially visible (e.g., under occlusions). In contrast, we propose a deep-learning framework, PA-Tran, that jointly estimates the keypoints status and 3D hand pose from a single RGB image with two dependent branches. The regression branch consists of a Transformer encoder which is trained to predict a set of target keypoints, given an input set of status, position, and visual features embedding from a convolutional neural network (CNN); the classification branch adopts a CNN for estimating the keypoints status. One key idea of PA-Tran is a selective mask training (SMT) objective that uses a binary encoding scheme to represent the status of the keypoints as observed or unobserved during training. In addition, by explicitly encoding the label status (observed/unobserved), the proposed PA-Tran can efficiently handle the condition when only partial annotation is available. Investigating the annotation percentage ranging from 50-100%, we show that training with partial annotation is more efficient (e.g., achieving the best 6.0 PA-MPJPE when using about 85% annotations). Moreover, we provide two new datasets. APDM-Hand, is for synthetic hands with APDM sensor accessories, which is designed for a specific hand task. PD-APDM-Hand, is a real hand dataset collected from Parkinson's Disease (PD) patients with partial annotation. The proposed PA-Tran can achieve higher estimation accuracy when evaluated on both proposed datasets and a more general hand dataset.

Keywords: 3D hand pose estimation; PD (Parkinson’s disease) hand dataset; partial annotation; single RGB image; synthetic dataset; transformer.

Figure 1

Biological characteristics of the human hand skeleton: (a) Illustration of the DoF of the hand; (b) Indices of the hand joints.

Figure 2

Overview of the proposed PA-Tran framework. Given an input image $\mathcal{I}$, we extract the image features using a convolution neural network. Then the image features are passed into two separate branches: the regression branch $reg(·)$ and the classification branch $cla(·)$. $cla(·)$ will generate the status embedding for $reg(·)$ and masks for $SMT$ to learn the interaction between labels. The structures of $reg(·)$ and $cla(·)$ are detailed in Section 3.2.

Figure 3

The structure of the $reg(·)$ branch. The input is the concatenation of feature embedding, position embedding, and status embedding. Sequential transformer blocks are adopted to reduce the dimension of the hidden embedding progressively. The final output is the coordinates of the keypoints.

Figure 4

Examples of finger-tapping animation frames with motion blur.

Figure 5

Examples of hand-movement animation frames with motion blur.

Figure 6

Examples of APDM-Hand images from different views and backgrounds.

Figure 7

Examples of PD-APDM-Hand, which is collected from real Parkinson’s Disease patients when taking the UPDRS test.

Figure 8

Qualititive result on APDM-Hand dataset: (a) Ground truth; (b) METRO; (c) PA-Tran.

Figure 9

Qualititive results on PD-APDM-Hand dataset: (a) PD subject 1; (b) PD subject 2.

Figure 10

Hand pose estimation with motion blur.