A circadian behavioral analysis suite for real-time classification of daily rhythms in complex behaviors

Abstract

Measuring animal behavior over long timescales has been traditionally limited to behaviors that are easily measurable with real-time sensors. More complex behaviors have been measured over time, but these approaches are considerably more challenging due to the intensive manual effort required for scoring behaviors. Recent advances in machine learning have introduced automated behavior analysis methods, but these often overlook long-term behavioral patterns and struggle with classification in varying environmental conditions. To address this, we developed a pipeline that enables continuous, parallel recording and acquisition of animal behavior for an indefinite duration. As part of this pipeline, we applied a recent breakthrough self-supervised computer vision model to reduce training bias and overfitting and to ensure classification robustness. Our system automatically classifies animal behaviors with a performance approaching that of expert-level human labelers. Critically, classification occurs continuously, across multiple animals, and in real time. As a proof-of-concept, we used our system to record behavior from 97 mice over two weeks to test the hypothesis that sex and estrogen influence circadian rhythms in nine distinct home cage behaviors. We discovered novel sex- and estrogen-dependent differences in circadian properties of several behaviors including digging and nesting rhythms. We present a generalized version of our pipeline and novel classification model, the "circadian behavioral analysis suite," (CBAS) as a user-friendly, open-source software package that allows researchers to automatically acquire and analyze behavioral rhythms with a throughput that rivals sensor-based methods, allowing for the temporal and circadian analysis of behaviors that were previously difficult or impossible to observe.

Figure 1.. Recording and classification standardization of nine home cage behaviors.

a) Schematic of the home cage recording setup. b) Representative examples of individual frames depicting each of nine behaviors (eating, orange; drinking, yellow; rearing, blue; climbing, red; grooming, green; exploring, brown; digging, magenta; nesting, purple; resting, gray). First frame depicts the behavior occurring in the full field of view, subsequent frames are zoomed in to better illustrate behaviors. c) Bout length (duration of a behavioral instance) for each behavior within a maximum window size of 360 s. n ≥ 38 bouts from 29 to 30 mice per behavior. Box and whiskers depict median and interquartile range. d) Number of unique instances of each behavior in the 8.1 h human-labeled dataset broken down by training and test sets.

Figure 2.. DINOv2+ approaches expert-level performance on behavior classification.

a) Schematic of performance and generalization tests. Features from a frozen pretrained DeepEthogram (DEG) model and a frozen pretrained DINOv2 model were used to evaluate the ability of each visual feature extractor to successfully classify mouse behavior using our DINOv2+ joint LSTM and linear layer model head (performance; Figs. 2b,c), classify mouse behavior on behavior frames rotated 90° using a single layer linear network head (generalization; Fig. 2d), and count the number of mice in a cage using a single layer linear network head (generalization; Fig. 2e). b) Precision-recall curves for each behavior calculated for the DINOv2+ (colored lines) and DEG (dashed lines) models by varying the decision threshold of each binary classifier. Shading depicts the area under the precision-recall curve (AUPRC) for each behavior for each model. Bootstrap test; **, p < 0.01; ***, p < 0.001. c) Performance metrics for each behavior calculated for a trained human classifier (green), the DEG model (blue), and the DINOv2+ model (red). n = 10 sets of 1,000 randomly sampled test set frames per behavior. Dashed line depicts a predefined performance threshold of 0.80. Lines and error bars depict mean ± SEM. F1, F1 score; nMCC, normalized Matthews correlation coefficient. d) Relative performance for the DEG (blue) and DINOv2 (red) pretrained models when tested on a rotated version of a baseline behavior sequence test set using a single layer linear network head on top of the baseline models. e) F1 score calculated for both DEG and DINOv2 on a classification task involving counting the number of mice in a cage using a single layer linear network head on top of the baseline models.

Figure 3.. DINOv2+ allows for real-time behavior classification.

a) Schematic of the real-time video recording, processing, and inferring system comprising two sets of 12 PoE (power over ethernet) IP cameras networked to a switch that passes streaming video data to a machine learning computer for video inference and a network-attached storage device for video backup. b) Single-video inference times for video segments of various lengths calculated for a skeletal pose estimation model without behavior classification (green, DLC), DEG (blue), and DINOv2+ (red). n = 3 replicates per model. Two-way ANOVA with post-hoc Tukey’s multiple comparison’s test; *, p < 0.05; ***, p < 0.001. c) Inference times for combinations of video segment length and number of cameras used to simultaneously stream video segments calculated for each model. Dashed lines depict the times at which inference time equals the length of the video segment. Failure of real-time inference for a particular combination of segment length, camera number, and inference model is represented by a black X above the bar. d) Representative activity profiles for each behavior from an individual mouse recorded in a 12 h:12 h light:dark (LD) cycle for 48 h. 30 min segments of continuously recorded video were automatically processed, inferred, and plotted over the duration of the recording, “filling in” over time. For visualization, plots shown here are only updated every 6 h. ZT, zeitgeber time.

Figure 4.. Male and female mice exhibit distinct circadian rhythms in home cage behaviors.

a,b) Representative double-plotted actograms depicting behaviors (colored lines on each row) averaged across eight male mice or eight female mice that started the experiment in the same estrous state recorded over 5 d in a 12h:12h light:dark (LD) cycle (gray and yellow shading) and 9 d in constant darkness (DD; gray and light gray shading). c) Behavior phase comparison plots depicting the acrophases (peak times in circadian time, where CT 18 is subjective midnight and CT 6 is subjective noon) for male (teal, n = 24), metestrus/diestrus (M/D; pink), and proestrus/estrus (P/E; purple) female (n = 27) mice recorded in DD. Lines and error bars depict mean ± SEM. Asterisks indicate behaviors with significant differences in acrophase across groups. One-way ANOVA with post-hoc Tukey’s multiple comparisons test; *, p < 0.05; **, p < 0.01. d) Normalized amplitude for each behavior rhythm for male (teal), M/D (pink), and P/E (purple) female mice measured in DD. Two-way ANOVA with post-hoc Tukey’s multiple comparisons test; *, p < 0.05; **, p < 0.01; ***, p < 0.001.

Figure 5.. Ovariectomized and ovariectomized, estradiol-supplemented female mice exhibit distinct circadian rhythms in home cage behaviors.

a,b) Representative double-plotted actograms depicting behaviors (colored lines on each row) averaged across eight ovariectomized (OVX) female mice or eight ovariectomized, estradiol-supplemented (OVXE) female mice recorded over 5 d in a 12h:12h light:dark (LD) cycle (gray and yellow shading) and 5 d in constant darkness (DD; gray and light gray shading). c) Behavior phase comparison plots depicting the acrophases (peak times in circadian time, where CT 18 is subjective midnight and CT 6 is subjective noon) for OVX (pink, n = 24) and OVXE (purple; n = 22) female mice recorded in DD. Lines and error bars depict mean ± SEM. Asterisks indicate behaviors with significant differences in acrophase across groups. One-way ANOVA with post-hoc Tukey’s multiple comparisons test; *, p < 0.05; **, p < 0.01. d) Normalized amplitude for each behavior rhythm for OVX (pink) and OVXE (purple) female mice measured in DD. Two-way ANOVA with post-hoc Tukey’s multiple comparisons test; **, p < 0.01.

Figure 6.. CBAS: a circadian behavioral analysis suite.

a) CBAS is a user-friendly GUI-enabled Python package that allows for the automated acquisition, classification, and visualization of behaviors over time. b) Schematic of the CBAS pipeline. Red; acquisition module; blue, training module; green, classification and visualization classification module.