LarvaTagger: manual and automatic tagging of Drosophila larval behaviour

Abstract

Motivation: As more behavioural assays are carried out in large-scale experiments on Drosophila larvae, the definitions of the archetypal actions of a larva are regularly refined. In addition, video recording and tracking technologies constantly evolve. Consequently, automatic tagging tools for Drosophila larval behaviour must be retrained to learn new representations from new data. However, existing tools cannot transfer knowledge from large amounts of previously accumulated data. We introduce LarvaTagger, a piece of software that combines a pre-trained deep neural network, providing a continuous latent representation of larva actions for stereotypical behaviour identification, with a graphical user interface to manually tag the behaviour and train new automatic taggers with the updated ground truth.

Results: We reproduced results from an automatic tagger with high accuracy, and we demonstrated that pre-training on large databases accelerates the training of a new tagger, achieving similar prediction accuracy using less data.

Availability and implementation: All the code is free and open source. Docker images are also available. See gitlab.pasteur.fr/nyx/LarvaTagger.jl.

Figure 1.

(a) Behaviour analysis pipelines. Behaving larvae are video-grabbed and tracked, which results in trajectory data together with shape data at each time point of a trajectory. The analysis of such data typically relies on identifying actions from a dictionary of actions, again at each time step of a larva’s trajectory. The resulting discrete behavioural readout can be compared between populations of larvae in terms of action proportions or sequences (top-right plot of action probability versus time). Alternatively, tracking data can be projected into a common feature space, with track segments represented as points. The feature space, also referred to as latent space (bottom-centre illustration of time segments as points in a 2D latent space), is first generated in an unsupervised fashion using large amounts of unlabelled data. Such representations can be used to design statistical tests or any analysis that involves groups or categories. As an example, transition probabilities between two actions are illustrated in a 2D latent space. LarvaTagger implements both action identification and generation of latent representations using a MaggotUBA-based tagger. (b) Integration of MaggotUBA (top diagram), an autoencoder that extracts features of the continuous behaviour in a self-supervised fashion. A sequence of postures X_t (or track segment) is compressed into a low-dimensional latent representation Z from which a longer sequence, including past and future postures in addition to the input X_t, is reconstructed. Training the autoencoder allows learning behavioural features that compress the dynamics in the latent space. The encoder features are reused in combination with a classification stage in the MaggotUBA-based tagger (bottom diagram) to learn a discrete behavioural dictionary. The MaggotUBA-based tagger that LarvaTagger embarks can be used to generate both types of behavioural readouts.

Figure 2.

Left panels originate from Pipeline_pasteur_janelia. Right panels originate from 20230311. (a) and (b) Probability time series for the six actions of interest (small action not accounted for) in a population of control larvae (w;;attP2). At $t=45s$, the larvae received a 38-s long air puff, resulting in a dramatic change in behaviour reflected in the action probabilities over time. While baseline probabilities are well preserved in (b), as compared with (a), short-term response probabilities exhibit slightly more overall changes in (b), with more frequent hunches, back-ups and stops. (c) and (d) Significant differences in individual action probabilities between a selection of Drosophila lines and the control w;;attP2 line. All P-values are Bonferroni-corrected for 471 comparisons. (c) is a reproduction of Fig. 3e in Masson et al. (2020). Some differences between (c) and (d) can be observed. In particular, a few effects in (c) are lost in (d), and, more frequently, effects observed with the 20230311 tagger (d) were not found using the original tagger (c).

Figure 3.

f1-scores for different training dataset sizes (abscissa), with the MaggotUBA encoder pre-trained (“True” curve) or not (“False” curve). Train and test datasets were drawn from the new optogenetic screen. Pre-training was performed on Jovanic et al. (2016) and Masson et al. (2020).