Robust fetoscopic mosaicking from deep learned flow fields

Abstract

Purpose: Fetoscopic laser photocoagulation is a minimally invasive procedure to treat twin-to-twin transfusion syndrome during pregnancy by stopping irregular blood flow in the placenta. Building an image mosaic of the placenta and its network of vessels could assist surgeons to navigate in the challenging fetoscopic environment during the procedure.

Methodology: We propose a fetoscopic mosaicking approach by combining deep learning-based optical flow with robust estimation for filtering inconsistent motions that occurs due to floating particles and specularities. While the current state of the art for fetoscopic mosaicking relies on clearly visible vessels for registration, our approach overcomes this limitation by considering the motion of all consistent pixels within consecutive frames. We also overcome the challenges in applying off-the-shelf optical flow to fetoscopic mosaicking through the use of robust estimation and local refinement.

Results: We compare our proposed method against the state-of-the-art vessel-based and optical flow-based image registration methods, and robust estimation alternatives. We also compare our proposed pipeline using different optical flow and robust estimation alternatives.

Conclusions: Through analysis of our results, we show that our method outperforms both the vessel-based state of the art and LK, noticeably when vessels are either poorly visible or too thin to be reliably identified. Our approach is thus able to build consistent placental vessel mosaics in challenging cases where currently available alternatives fail.

Keywords: Fetoscopy; Optical flow; Twin-to-twin transfusion syndrome; Video mosaicking.

Fig. 1

An overview of the proposed framework which is composed of a flow field generation block that provides features, a pixel correspondence block that performs feature matching, and a registration block that generates mosaic through LM optimization

Fig. 2

Quantitative comparison of the proposed (red), vessel segmentation-based (blue), RAFT backbone (green) and LK-based (purple), RR (light purple) methods using the drift analysis metric from [4]

Fig. 3

Visualization of the mosaics produced by our proposed method. The first column shows final mosaics on various sequences from the extended dataset using our method while the second column show final mosaics on the same sequences using the state-of-the-art approach from [4]. The third column shows SSIM time plots which plot the SSIM of the registration of consecutive images. The red colored plot is our method while the blue is the baseline (Table 2). Notice that the tracking fails in Videos 3, 4, 6 in the case of [4] after frame number 113, 106, 140. While our proposed method resulted in consistent mosaics for the complete duration of all extended videos

Fig. 4

Vessel-based method qualitative analysis using Video 4. Top row shows when vessel segmentation method fails in registration (at frame 106). Bottom row shows vessel segmentation with good registration(at frame 99 highest SSIM). From left to right : destination image, destination image vessels segmentation, source image, source image vessels segmentation, registration using vessel segmentation, registration using our method

Fig. 5

Examples of outlier regions (in red) detected by RANSAC. Outliers generally correspond to floating particles and bright specular reflections(white and bright spots on the Image) inconsistent with fetoscope motion. Images obtained from Video 1 (top-left), Video 2 (top-right), Video 5 (bottom-left), Video 6 (bottom-right)