3D mouse pose from single-view video and a new dataset

Abstract

We present a method to infer the 3D pose of mice, including the limbs and feet, from monocular videos. Many human clinical conditions and their corresponding animal models result in abnormal motion, and accurately measuring 3D motion at scale offers insights into health. The 3D poses improve classification of health-related attributes over 2D representations. The inferred poses are accurate enough to estimate stride length even when the feet are mostly occluded. This method could be applied as part of a continuous monitoring system to non-invasively measure animal health, as demonstrated by its use in successfully classifying animals based on age and genotype. We introduce the Mouse Pose Analysis Dataset, the first large scale video dataset of lab mice in their home cage with ground truth keypoint and behavior labels. The dataset also contains high resolution mouse CT scans, which we use to build the shape models for 3D pose reconstruction.

Figure 1

Left: The 2D keypoint names and corresponding color-coded markers shown in the labeling interface. Center: A labeled image of a mouse with the keypoint legends to the left. Right: The high resolution CT scan segmented for bone in light colors, and segmented for the skin in darker colors with the corresponding keypoint locations at a neutral pose.

Figure 2

A heatmap of all annotated mouse keypoints displayed in the home cage. Each dot represents one keypoint. Majority of the activities happen on the wheel and near the feeder.

Figure 3

Top: Pipeline diagram. Rectangular boxes are algorithms and processes. Ellipses are intermediate and final results of the pipeline. Bottom:Pictorial depiction of the pipeline. It operates over frames of a video (left panel). For each frame we run a 2D object detector trained to detect mice (second panel, box indicating a detection). We apply a 2D pose model to detect mouse keypoints at the detected location (third panel, colored heatmap indicating joint locations with arbitrary colors). Finally, we optimize for the 3D pose of the mouse (right panel, blue points are peaks of the keypoint heatmaps in previous stage, red points are projected 3D keypoints from the optimized pose, grey 3D mesh overlaid on the image).

Figure 4

Comparison of multi-view and single-view reconstructions. The error bars are $\pm 1$ SE. The top three panels show three views of the mouse at the same time point. Red dots are reconstructions from triangulation and cyan dots from our single-view reconstruction. Four of 20 joints are shown as examples (0: tail, 1: noise, 2: left paw and 3: right paw).

Figure 5

Top left: An example time series of the foot position in arbitrary units. The periodic structure of gait is clearly visible. Red dots indicate peaks used in computing the stride length. Top right: The peak frequency in the foot position reconstruction $\times$ belt speed (blue, solid) and DigiGait posture plot stride length (orange, dashed). Bottom left: The distribution of stride lengths from the pose reconstruction (dark blue) and DigiGait (light orange). Dashed, black, vertical lines indicate outlier thresholds for statistical modeling. Bottom right: Stride lengths by treadmill speed for reconstructed pose (blue, solid) and DigiGait (orange, dashed). Error bars indicate ±1 SEM.

Figure 6

Representative confusion matrix for behavior classification. Each row represents the predicted classification for a given true positive label. Each column is a different output prediction. This particular confusion matrix is for the Images model, but the pattern is consistent across input types.