WormSwin: Instance segmentation of C. elegans using vision transformer

Abstract

The possibility to extract motion of a single organism from video recordings at a large-scale provides means for the quantitative study of its behavior, both individual and collective. This task is particularly difficult for organisms that interact with one another, overlap, and occlude parts of their bodies in the recording. Here we propose WormSwin-an approach to extract single animal postures of Caenorhabditis elegans (C. elegans) from recordings of many organisms in a single microscope well. Based on transformer neural network architecture our method segments individual worms across a range of videos and images generated in different labs. Our solutions offers accuracy of 0.990 average precision ([Formula: see text]) and comparable results on the benchmark image dataset BBBC010. Finally, it allows to segment challenging overlapping postures of mating worms with an accuracy sufficient to track the organisms with a simple tracking heuristic. An accurate and efficient method for C. elegans segmentation opens up new opportunities for studying of its behaviors previously inaccessible due to the difficulty in the worm extraction from the video frames.

Figure 1

Example images from the datasets used in this study: (a) synthetic dataset example with added ring, (b) synthetic dataset without ring, (c) BBBC010 dataset example with mostly alive C. elegans, (d) BBBC010 dataset patch with mostly dead C. elegans, (e) mating dataset with petri-dish ring, (f) zoomed-in mating dataset patch with many overlaps.

Figure 2

Network architecture based on Swin-L backbone and HTC. Batch norm (BN) layers in HTC are replaced by group norm (GN) + weight standardization (WS). Bounding box heads are changed from the original Shared2FC architecture to Shared4Conv1FC.

Figure 3

Example from the CSB-1 dataset (box and mask colors are selected randomly). (a) Ground truth annotations, (b) predicted bounding boxes and masks, (c) TP (green), FP and FN (red) pixels.

Figure 4

Results on the Mating Dataset (box and mask colors are selected randomly). (a,c,e,g) Segmentation results, (b,d,f,h) TP (green), FP and FN (red) pixels.

Figure 5

Example of tracked C. elegans.