Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Sep 18;44(5):967-980.
doi: 10.24272/j.issn.2095-8137.2022.449.

Deep learning-based activity recognition and fine motor identification using 2D skeletons of cynomolgus monkeys

Chuxi Li  1 , Zifan Xiao  2   3   4 , Yerong Li  1 , Zhinan Chen  1 , Xun Ji  5 , Yiqun Liu  6 , Shufei Feng  7   8 , Zhen Zhang  7   8 , Kaiming Zhang  9 , Jianfeng Feng  2   3   4 , Trevor W Robbins  2   3   4   10 , Shisheng Xiong  11 , Yongchang Chen  7   8 , Xiao Xiao  2   3   12
Affiliations

Deep learning-based activity recognition and fine motor identification using 2D skeletons of cynomolgus monkeys

Chuxi Li et al. Zool Res. .

Abstract
in English, Chinese

Video-based action recognition is becoming a vital tool in clinical research and neuroscientific study for disorder detection and prediction. However, action recognition currently used in non-human primate (NHP) research relies heavily on intense manual labor and lacks standardized assessment. In this work, we established two standard benchmark datasets of NHPs in the laboratory: MonkeyinLab (MiL), which includes 13 categories of actions and postures, and MiL2D, which includes sequences of two-dimensional (2D) skeleton features. Furthermore, based on recent methodological advances in deep learning and skeleton visualization, we introduced the MonkeyMonitorKit (MonKit) toolbox for automatic action recognition, posture estimation, and identification of fine motor activity in monkeys. Using the datasets and MonKit, we evaluated the daily behaviors of wild-type cynomolgus monkeys within their home cages and experimental environments and compared these observations with the behaviors exhibited by cynomolgus monkeys possessing mutations in the MECP2 gene as a disease model of Rett syndrome (RTT). MonKit was used to assess motor function, stereotyped behaviors, and depressive phenotypes, with the outcomes compared with human manual detection. MonKit established consistent criteria for identifying behavior in NHPs with high accuracy and efficiency, thus providing a novel and comprehensive tool for assessing phenotypic behavior in monkeys.

在神经科学及临床研究中，基于视频的动作识别方法日益成为神经疾病检测和预测的重要工具。然而，非人灵长类动物研究的动作识别目前仍依赖高强度的人工操作，并且缺乏标准化的评估方法，极大地影响研究效率及识别准确性。因此，该研究建立了在实验室环境下的两个非人灵长类动物基准数据集：MonkeyinLab (MiL) 数据集（包括13 类动作和姿态）和MiL2D数据集（包括15个2D 骨骼特征点），这两个标准数据集覆盖了食蟹猴的日常表型行为。此外，该研究还提出了一个基于深度学习的工具箱MonkeyMonitorKit (MonKit)，MonKit采用TSSA网络识别猴子动作（准确度98.99%）和HRNet网络识别猴子的骨骼特征点（准确度98.8%），并结合动作姿势估计建立了精细动作的评估模型和行为分析方法，可评估疾病状态下的低头行为（抑郁症样表型）、刻板行为（自闭症样表型）等。该研究利用MonKit工具箱量化比较了作为 Rett 综合征疾病模型的MECP2 基因突变食蟹猴与野生型食蟹猴的日常行为分类和精细动作评估，并与人工识别进行对比，确认MonKit工具箱具备自动化、高效、标准统一、高准确率、高敏感性和低误差等优势，为非人灵长类的表型行为分类和评估提供了一种新颖而全面的研究手段。.

Keywords: 2D skeleton; Action recognition; Fine motor identification; Non-human primates; Rett syndrome.; Two-stream deep model.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Examples of MiL dataset, with videos corresponding to actions and postures (labels)
Figure 2
Figure 2
Two-stream action recognition model and heatmaps of feature visualization
Figure 3
Figure 3
Action recognition of daily performance in WT monkeys in home cage, WT monkeys in test cage, and MECP2 mutant monkeys in test cage
Figure 4
Figure 4
Illustrations of MiL2D dataset with 15 skeleton keypoints
Figure 5
Figure 5
MonKit for action tracking and posture estimation based on MaskTrack R-CNN and HRNet
Figure 6
Figure 6
Posture recognition detected by MonKit in WT monkeys in home cage, WT monkeys in test cage, and MECP2 mutant monkeys
Figure 7
Figure 7
Stereotyped behavior patterns and head-down posture detected by MonKit
Figure 8
Figure 8
Comparison of MonKit and manual detection accuracy
Figure 9
Figure 9
Schematic of datasets and MonKit
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Ahmad Z, Khan N Human action recognition using deep multilevel multimodal (M2) fusion of depth and inertial sensors. IEEE Sensors Journal. 2020;20(3):1445–1455. doi: 10.1109/JSEN.2019.2947446. - DOI
    1. Amir RE, Van den Veyver IB, Wan MM, et al Rett syndrome is caused by mutations in X-linked MECP2, encoding methyl-CpG-binding protein 2. Nature Genetics. 1999;23(2):185–188. doi: 10.1038/13810. - DOI - PubMed

    1. Andriluka M, Pishchulin L, Gehler P, et al. 2014. 2D human pose estimation: new benchmark and state of the art analysis. In: Proceedings of 2014 IEEE Computer Vision and Pattern Recognition. Columbus: IEEE.

    1. Bala PC, Eisenreich BR, Yoo SBM, et al Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio. Nature Communications. 2020;11(1):4560. doi: 10.1038/s41467-020-18441-5. - DOI - PMC - PubMed
    1. Ben Mabrouk A, Zagrouba E Abnormal behavior recognition for intelligent video surveillance systems: a review. Expert Systems with Applications. 2018;91:480–491. doi: 10.1016/j.eswa.2017.09.029. - DOI