Fast and robust optical flow for time-lapse microscopy using super-voxels

Abstract

Motivation: Optical flow is a key method used for quantitative motion estimation of biological structures in light microscopy. It has also been used as a key module in segmentation and tracking systems and is considered a mature technology in the field of computer vision. However, most of the research focused on 2D natural images, which are small in size and rich in edges and texture information. In contrast, 3D time-lapse recordings of biological specimens comprise up to several terabytes of image data and often exhibit complex object dynamics as well as blurring due to the point-spread-function of the microscope. Thus, new approaches to optical flow are required to improve performance for such data.

Results: We solve optical flow in large 3D time-lapse microscopy datasets by defining a Markov random field (MRF) over super-voxels in the foreground and applying motion smoothness constraints between super-voxels instead of voxel-wise. This model is tailored to the specific characteristics of light microscopy datasets: super-voxels help registration in textureless areas, the MRF over super-voxels efficiently propagates motion information between neighboring cells and the background subtraction and super-voxels reduce the dimensionality of the problem by an order of magnitude. We validate our approach on large 3D time-lapse datasets of Drosophila and zebrafish development by analyzing cell motion patterns. We show that our approach is, on average, 10 × faster than commonly used optical flow implementations in the Insight Tool-Kit (ITK) and reduces the average flow end point error by 50% in regions with complex dynamic processes, such as cell divisions.

Availability: Source code freely available in the Software section at http://janelia.org/lab/keller-lab.

Fig. 1.

(A) Rendering of 3D volume obtained with SiMView light-sheet microscopy (Tomer et al., 2012). Each of the objects represents a single cell nucleus marked by a fluorescent reporter in a Drosophila embryo. Dimensions are 602 × 1386 × 110 voxels per volume (0.4 × 0.4 × 2.0 µm³ voxel size). The embryo is ∼550 µm long and 200 µm in diameter. (B) Optical slices of the volume visualized in (A). (C) Enlarged view of two superimposed consecutive time points. Multiple motions, such as cell divisions and cell migration, occur in the same volume

Fig. 2.

Block diagram representing the pipeline described in this article to estimate optical flow. Optical flow is performed over a set of super-voxels in the volume foreground, and the smoothness constraints are imposed between neighboring (and possibly non-adjacent) super-voxels instead of between connected voxels. This approach guides the registration process of neighboring nuclei with similar dynamics to a better solution than previous approaches

Fig. 3.

Step for constructing an MRF over the super-voxels on the volume foreground to partition the volume and perform robust optical flow. (A) 2D slice of raw data from Figure 1. We show only a slice to simplify the visualization, but the method is implemented in 3D. (B) Outline of the foreground mask obtained with a trained classifier in Ilastik (Sommer et al., 2011). Some connected components correspond to multiple nuclei. (C) Slice of 3D SLIC (Achanta et al., 2012) super-voxels calculated over the foreground. Super-voxels respect object boundaries of nuclei in the same foreground connected component. (D) Edges added between neighboring super-voxels to generate an MRF. Each node V_i represents a super-voxel in panel C. This is the final volume partition model where we perform optical flow. We impose the smoothness conditions over entire super-voxels instead of voxelwise

Fig. 4.

(A) Motion field (black: ground truth, red: estimate by our approach) projected on the X–Y plane for a subregion of the volume in Figure 1 with smooth flow. Each arrow corresponds to a nucleus centroid. (B) Same as (A) for motion field estimated by the baseline method multi-scale ITK-demon (blue). (C) Enlarged subregion of (A) and (B). (D) Same as (A) for a subregion where cells are dividing, which translates into non-smooth dynamics for neighboring nuclei. Our approach is still able to predict the correct motion for 99% of the nuclei. Supplementary Movie S1 shows the raw data and the output of our optical flow algorithm side by side for the entire time series. (E) Same as (B) for the subregion presented in (D). The complex dynamics complicate setting a global motion smoothing parameter that works for all nuclei at the same time. (F) Enlarged subregion of (D) and (E). Most of the ITK flow (blue) results as zero because it cannot adapt to the complex motion pattern

Fig. 5.

Optical flow results for light-sheet microscopy using different methods. See text for details on ground truth definition. X-axis represents the EE for each nucleus centroid normalized by the equivalent diameter of each nucleus. As a rule of thumb, values < 0.5 are considered good for most quantitative applications, whereas values > 1.0 are not good. Values between 0.5 and 1.0 are acceptable, but flow tracking has a higher error rate. Method labeled as ‘None’ represents the original displacement without flow estimation. Panel A shows results on data from Figure 4A and B. Panel B shows results on data from Figure 4D and E. Our method improves accuracy over all baselines in both scenarios, on average, by 23%