Development of Smartphone Application for Markerless Three-Dimensional Motion Capture Based on Deep Learning Model

Abstract

To quantitatively assess pathological gait, we developed a novel smartphone application for full-body human motion tracking in real time from markerless video-based images using a smartphone monocular camera and deep learning. As training data for deep learning, the original three-dimensional (3D) dataset comprising more than 1 million captured images from the 3D motion of 90 humanoid characters and the two-dimensional dataset of COCO 2017 were prepared. The 3D heatmap offset data consisting of 28 × 28 × 28 blocks with three red-green-blue colors at the 24 key points of the entire body motion were learned using the convolutional neural network, modified ResNet34. At each key point, the hottest spot deviating from the center of the cell was learned using the tanh function. Our new iOS application could detect the relative tri-axial coordinates of the 24 whole-body key points centered on the navel in real time without any markers for motion capture. By using the relative coordinates, the 3D angles of the neck, lumbar, bilateral hip, knee, and ankle joints were estimated. Any human motion could be quantitatively and easily assessed using a new smartphone application named Three-Dimensional Pose Tracker for Gait Test (TDPT-GT) without any body markers or multipoint cameras.

Keywords: deep learning; markerless motion capture; motion tracking; quantitative gait assessment; smartphone device.

Figure 2

Overview of the training process and backbone network for the pose estimation framework. The colored cells are the basic cells designed using ResNet34, and the white cells are the added cells.

Figure 1

Sample of the input training data for deep learning. The three consecutive images on the left (black 1–3) are dancing movements, and those on the right (white 1–3) are walking movements.

Figure 3

Output 3D relative coordinates of 24 key points on 3D heatmaps.

Figure 4

Snapshots of the TDPT for Gait Test (TDPT-GT) application and calculated 3D joint angles and artificial intelligence (AI) scores, also called the confidence scores in a healthy young volunteer. The iPhone was fixed as horizontal as possible to the floor. The entire body of the subject from the head to the toes must always be in the videoframe. The blue-colored number at the left of the videoframe shows the reliability of the position information of the entire body as an AI score, and the small numbers below show the 3D angles (AI scores) of the left and right knee joints, and (AI scores) of the left and right ankle joints.

Figure 5

Three-dimensional angles (solid lines) calculated by the relative 3D coordinates estimated by the TDPT for Gait Test (TDTP-GT) application and artificial intelligence (AI) scores (dotted lines). When each AI score is 0.7 (black dotted line) or higher, the 3D angles at the right (blue) and left (red) hip joints (a), knee joints (b), and ankle joints (c) are considered to be relatively reliable.

Figure 6

Simultaneous measurement by two methods of TDPT for Gait Test (TDPT-GT) application and Vicon Motion System. Blue dots indicate 3D coordinates; blue lines indicate both upper extremities, yellow indicates the trunk, gray indicates the left lower extremity, and orange indicates the right lower extremity.