Multimodal video and IMU kinematic dataset on daily life activities using affordable devices

Abstract

Human activity recognition and clinical biomechanics are challenging problems in physical telerehabilitation medicine. However, most publicly available datasets on human body movements cannot be used to study both problems in an out-of-the-lab movement acquisition setting. The objective of the VIDIMU dataset is to pave the way towards affordable patient gross motor tracking solutions for daily life activities recognition and kinematic analysis. The dataset includes 13 activities registered using a commodity camera and five inertial sensors. The video recordings were acquired in 54 subjects, of which 16 also had simultaneous recordings of inertial sensors. The novelty of dataset lies in: (i) the clinical relevance of the chosen movements, (ii) the combined utilization of affordable video and custom sensors, and (iii) the implementation of state-of-the-art tools for multimodal data processing of 3D body pose tracking and motion reconstruction in a musculoskeletal model from inertial data. The validation confirms that a minimally disturbing acquisition protocol, performed according to real-life conditions can provide a comprehensive picture of human joint angles during daily life activities.

Fig. 1

Examples of data files included in the dataset for upper (left) and lower body activities (right): a raw video file, a pose estimator from video, and a 3D motion reconstruction from inertial data. For upper-body movements (left), the subjects wear 5 IMUs in the upper limbs. For lower-body movements (right), the subjects wear 5 IMUs on the lower limbs. Individuals in the figures provided consent for their images to be published.

Fig. 2

Lower limb activities in the VIDIMU dataset. Individuals in the figures provided consent for their images to be published.

Fig. 3

Upper limb activities in the VIDIMU dataset. Individuals in the figures provided consent for their images to be published.

Fig. 4

IMU sensor’s reference system and sensors placement. Individual in the figure provided consent for his image to be published.

Fig. 5

Example results of synchronization processing by minimizing RMSE between IMU (red) and video (blue) sources of data. The X axis represents the signal sample, and the Y axis represents the joint angle. For a given activity (A01 in the figure), we show the reconstructed angle with IMU data (A) and with video data (B). Panel C shows the first 180 samples (6 seconds) of the same signals after median-filtered smoothing. Panel D shows the effect of mean removal, so that the absolute ranges of motion of joint angles estimated with both sources of data can be better compared. Finally, panel E represents the optimal shifting of one of the signals, so that the Root Mean Squared Error (RMSE) is minimized. The number of samples required to shift the video or IMU signals to the left is indicated in brackets on top of the subplot in panel E: e.g. “cut imu:0, cut vid:3” would mean that the video-derived signal needs to be shifted 3 samples to the left to be synchronized with the IMU-derived signal.

Fig. 6

Examples of estimated joint angles inferred from 3D joint positions for activity A01 and activity A10. From left to right: right shoulder, left shoulder, right elbow, left elbow, right knee, left knee. Equivalent plots for every subject and activity are included as dataset files.

Fig. 7

Examples of raw quaternion data collected for lower body activity A02 and upper body activity A05. For lower body activities, from left to right quaternion data from IMU sensors placed on hips, right upper leg, right lower leg, left upper leg and left lower leg. For upper body activities, from left to right quaternion data from IMU sensors placed on back, right upper arm, right lower arm, left upper arm, left lower arm. Equivalent plots for every subject and activity are included as dataset files.

Fig. 8

Examples of estimated joint angles computed through inverse kinematics from raw IMU data, for lower body activity A04 and upper body activity A07. Equivalent plots for every subject and activity including additional joint angles are included as dataset files. For lower body activities, these files include joint angles for: pelvis_tilt, pelvis_list, pelvis_rotation, hip_flexion_r, hip_adduction_r, hip_rotation_r, knee_angle_r, hip_flexion_l, hip_adduction_l, hip_rotation_l, knee_angle_l. For upperbody activities, they include joint angles for: lumbar_extension, lumbar_bending, lumbar_rotation, arm_flex_r, arm_add_r, arm_rot_r, elbow_flex_r, pro_sup_r, arm_flex_l, elbow_flex_l, pro_sup_l.

Fig. 9

Reconstruction of movements using inverse kinematics in OpenSim for subject S40. Left: lower body activities A01, A02, A03, and A04. Right: upper body activities A05, A09, A10, A13. Motion files (.mot) for every subject and activity are included as dataset files.