Marker-less tracking system for multiple mice using Mask R-CNN

Abstract

Although the appropriate evaluation of mouse behavior is crucial in pharmacological research, most current methods focus on single mouse behavior under light conditions, owing to the limitations of human observation and experimental tools. In this study, we aimed to develop a novel marker-less tracking method for multiple mice with top-view videos using deep-learning-based techniques. The following stepwise method was introduced: (i) detection of mouse contours, (ii) assignment of identifiers (IDs) to each mouse, and (iii) correction of mis-predictions. The behavior of C57BL/6 mice was recorded in an open-field arena, and the mouse contours were manually annotated for hundreds of frame images. Then, we trained the mask regional convolutional neural network (Mask R-CNN) with all annotated images. The mouse contours predicted by the trained model in each frame were assigned to IDs by calculating the similarities of every mouse pair between frames. After assigning IDs, correction steps were applied to remove the predictive errors semi-automatically. The established method could accurately predict two to four mice for first-look videos recorded under light conditions. The method could also be applied to videos recorded under dark conditions, extending our ability to accurately observe and analyze the sociality of nocturnal mice. This technology would enable a new approach to understand mouse sociality and advance the pharmacological research.

Keywords: Mask R-CNN; mouse behavior; multi-rodent tracking; psychiatric disorders; translational research.

FIGURE 1

Schematic flow of the proposed method. Mouse contours in each frame were independently identified (detection step). Every mouse was assigned IDs by calculating of similarities (tracking step). Finally, the prediction was semi-automatically corrected (correction step). The background of images for mouse contour regions were removed for visibility.

FIGURE 2

Detection of mouse contours. (A) Representative annotated images. Red points and yellow lines indicate vertices and edges of contour. (B) Schematic flow of training mask regional convolutional neural network (Mask R-CNN). RDP: Ramer–Douglas–Peucker algorithm. (C) Representative images of contour detection. The left, middle and right image show the contours predicted by the tentative detection model, the human-corrected contours, and the contours predicted by the final detection model, respectively. (D) The training loss values for the final detection model.

FIGURE 3

Assigning IDs to identified mice. (A) Schematic images of calculating the sum of intensities between two pixels. (B) Schematic images of the method to calculate the similarities. The number alongside the arrows indicates the averaged absolute differential values between pair’s histograms at N and N + 1 frame. Pairs that have first and second lowest values were assigned to same ID. The background of images for mouse contour regions were removed for visibility.

FIGURE 4

Evaluation of the proposed method. (A) Distance between geometric centers of mouse contours annotated by humans and those predicted by the proposed method. v5, v6, and v7 indicate the video no. 5, 6, and 7, respectively (see Supplementary Table 1). These videos were recorded under light conditions. #0, #1, #2, and #3 indicate the individual mouse IDs. (B) Cumulative traveled distances of individual mice in the video no. 7. (C) Distances between #0 mouse and other mice in the video no. 7. (D) Cumulative proximity time between #0 mouse and other mice in video no 7. We defined “proximate” when distance between geometric centers of each pair was less than 6 cm.