Marker Data Enhancement For Markerless Motion Capture

Abstract

Objective: Human pose estimation models can measure movement from videos at a large scale and low cost; however, open-source pose estimation models typically detect only sparse keypoints, which leads to inaccurate joint kinematics. OpenCap, a freely available service for researchers to measure movement from videos, addresses this issue using a deep learning model-the marker enhancer-that transforms sparse keypoints into dense anatomical markers. However, OpenCap performs poorly on movements not included in the training data. Here, we create a much larger and more diverse training dataset and develop a more accurate and generalizable marker enhancer.

Methods: We compiled marker-based motion capture data from 1176 subjects and synthesized 1433 hours of keypoints and anatomical markers to train the marker enhancer. We evaluated its accuracy in computing kinematics using both benchmark movement videos and synthetic data representing unseen, diverse movements.

Results: The marker enhancer improved kinematic accuracy on benchmark movements (mean error: 4.1°, max: 8.7°) compared to using video keypoints (mean: 9.6°, max: 43.1°) and OpenCap's original enhancer (mean: 5.3°, max: 11.5°). It also better generalized to unseen, diverse movements (mean: 4.1°, max: 6.7°) than OpenCap's original enhancer (mean: 40.4°, max: 252.0°).

Conclusion: Our marker enhancer demonstrates both accuracy and generalizability across diverse movements.

Significance: We integrated the marker enhancer into OpenCap, thereby offering its thousands of users more accurate measurements across a broader range of movements.

Keywords: Deep learning; markerless motion capture; musculoskeletal modeling and simulation; pose estimation; trajectory optimization.

Fig. 1.

The marker enhancer model predicts the 3D position of 43 anatomical markers (colored dots on right skeletons) from 20 video keypoint (colored triangles on left skeleton) positions. It consists of two models: the arm model predicts the 3D position of eight arm-located anatomical markers from seven arm and shoulder keypoint positions (blue) and the body model predicts the 3D position of 35 anatomical markers located on the shoulder, torso, and lower-body from 15 shoulder and lower-body keypoint positions (orange). Both models include the subject’s height and weight as input, and all marker positions are expressed with respect to a root marker (the midpoint of the hip keypoints; black square on left skeleton).

Fig. 2.

Kinematics of diverse movements computed from different synthetic marker sets: reference anatomical markers, video keypoints without and with noise, and anatomical markers predicted from noisy video keypoints using the marker enhancer from Uhlrich et al. (2023) [1] and our marker enhancer (LSTM) trained on the enhanced dataset. The instances shown are selected from the generalizability task dataset.