Review

. 2020 Oct 14;108(1):44-65.

doi: 10.1016/j.neuron.2020.09.017.

A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives

Alexander Mathis¹, Steffen Schneider², Jessy Lauer³, Mackenzie Weygandt Mathis⁴

Affiliations

¹ Center for Neuroprosthetics, Center for Intelligent Systems, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland; Brain Mind Institute, School of Life Sciences, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland; The Rowland Institute at Harvard, Harvard University, Cambridge, MA, USA. Electronic address: alexander.mathis@epfl.ch.
² The Rowland Institute at Harvard, Harvard University, Cambridge, MA, USA; University of Tübingen and International Max Planck Research School for Intelligent Systems, Tübingen, Germany.
³ Center for Neuroprosthetics, Center for Intelligent Systems, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland; Brain Mind Institute, School of Life Sciences, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland; The Rowland Institute at Harvard, Harvard University, Cambridge, MA, USA.
⁴ Center for Neuroprosthetics, Center for Intelligent Systems, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland; Brain Mind Institute, School of Life Sciences, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland; The Rowland Institute at Harvard, Harvard University, Cambridge, MA, USA. Electronic address: mackenzie.mathis@epfl.ch.

PMID: 33058765
DOI: 10.1016/j.neuron.2020.09.017

Free article

Review

A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives

Alexander Mathis et al. Neuron. 2020.

Free article

. 2020 Oct 14;108(1):44-65.

doi: 10.1016/j.neuron.2020.09.017.

Authors

Alexander Mathis¹, Steffen Schneider², Jessy Lauer³, Mackenzie Weygandt Mathis⁴

Affiliations

¹ Center for Neuroprosthetics, Center for Intelligent Systems, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland; Brain Mind Institute, School of Life Sciences, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland; The Rowland Institute at Harvard, Harvard University, Cambridge, MA, USA. Electronic address: alexander.mathis@epfl.ch.
² The Rowland Institute at Harvard, Harvard University, Cambridge, MA, USA; University of Tübingen and International Max Planck Research School for Intelligent Systems, Tübingen, Germany.
³ Center for Neuroprosthetics, Center for Intelligent Systems, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland; Brain Mind Institute, School of Life Sciences, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland; The Rowland Institute at Harvard, Harvard University, Cambridge, MA, USA.
⁴ Center for Neuroprosthetics, Center for Intelligent Systems, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland; Brain Mind Institute, School of Life Sciences, Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland; The Rowland Institute at Harvard, Harvard University, Cambridge, MA, USA. Electronic address: mackenzie.mathis@epfl.ch.

PMID: 33058765
DOI: 10.1016/j.neuron.2020.09.017

Abstract

Extracting behavioral measurements non-invasively from video is stymied by the fact that it is a hard computational problem. Recent advances in deep learning have tremendously advanced our ability to predict posture directly from videos, which has quickly impacted neuroscience and biology more broadly. In this primer, we review the budding field of motion capture with deep learning. In particular, we will discuss the principles of those novel algorithms, highlight their potential as well as pitfalls for experimentalists, and provide a glimpse into the future.

PubMed Disclaimer

Cited by

Perspectives on Individual Animal Identification from Biology and Computer Vision.
Vidal M, Wolf N, Rosenberg B, Harris BP, Mathis A. Vidal M, et al. Integr Comp Biol. 2021 Oct 4;61(3):900-916. doi: 10.1093/icb/icab107. Integr Comp Biol. 2021. PMID: 34050741 Free PMC article. Review.
Deep learning pose detection model for sow locomotion.
de Paula TMCG, de Sousa RV, Sarmiento MP, Kramer T, de Souza Sardinha EJ, Sabei L, Machado JS, Vilioti M, Zanella AJ. de Paula TMCG, et al. Sci Rep. 2024 Jul 16;14(1):16401. doi: 10.1038/s41598-024-62151-7. Sci Rep. 2024. PMID: 39013897 Free PMC article.
High-THC Cannabis Smoke Impairs Incidental Memory Capacity in Spontaneous Tests of Novelty Preference for Objects and Odors in Male Rats.
Barnard IL, Onofrychuk TJ, Toderash AD, Patel VN, Glass AE, Adrian JC, Laprairie RB, Howland JG. Barnard IL, et al. eNeuro. 2023 Dec 11;10(12):ENEURO.0115-23.2023. doi: 10.1523/ENEURO.0115-23.2023. Print 2023 Dec. eNeuro. 2023. PMID: 37973381 Free PMC article.
Machine Learning for Optical Motion Capture-Driven Musculoskeletal Modelling from Inertial Motion Capture Data.
Dasgupta A, Sharma R, Mishra C, Nagaraja VH. Dasgupta A, et al. Bioengineering (Basel). 2023 Apr 24;10(5):510. doi: 10.3390/bioengineering10050510. Bioengineering (Basel). 2023. PMID: 37237580 Free PMC article.
Receiver response to high-intensity courtship differs with courter status in spotted bowerbirds Ptilonorhynchus maculatus.
Spezie G, Mann DC, Knoester J, MacGillavry T, Fusani L. Spezie G, et al. R Soc Open Sci. 2024 Oct 23;11(10):232015. doi: 10.1098/rsos.232015. eCollection 2024 Oct. R Soc Open Sci. 2024. PMID: 39445089 Free PMC article.

See all "Cited by" articles

Publication types

Actions
Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
- Elsevier Science

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives

Affiliations

A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives

Authors

Affiliations

Abstract

Similar articles

Cited by

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources

Abstract

Similar articles

Cited by

Publication types

MeSH terms

Related information

LinkOut - more resources

Full Text Sources