Learning-based keypoint registration for fetoscopic mosaicking

Abstract

Purpose: In twin-to-twin transfusion syndrome (TTTS), abnormal vascular anastomoses in the monochorionic placenta can produce uneven blood flow between the two fetuses. In the current practice, TTTS is treated surgically by closing abnormal anastomoses using laser ablation. This surgery is minimally invasive and relies on fetoscopy. Limited field of view makes anastomosis identification a challenging task for the surgeon.

Methods: To tackle this challenge, we propose a learning-based framework for in vivo fetoscopy frame registration for field-of-view expansion. The novelties of this framework rely on a learning-based keypoint proposal network and an encoding strategy to filter (i) irrelevant keypoints based on fetoscopic semantic image segmentation and (ii) inconsistent homographies.

Results: We validate our framework on a dataset of six intraoperative sequences from six TTTS surgeries from six different women against the most recent state-of-the-art algorithm, which relies on the segmentation of placenta vessels.

Conclusion: The proposed framework achieves higher performance compared to the state of the art, paving the way for robust mosaicking to provide surgeons with context awareness during TTTS surgery.

Keywords: Deep learning; Fetal surgery; Fetoscopy; Mosaicking; Self-supervised; Twin-to-twin transfusion syndrome.

Fig. 1

Main challenges of TTTS frames: a occlusions, b, d, f lack of anatomical structures (e.g., vessels), c poor frame texture, e non-planar view in case of anterior placenta

Fig. 2

Overview of the semantic keypoint proposal network. KH and DH are the keypoint and keypoint descriptor head, respectively. Irrelevant keypoint rejection relies on semantic segmentation performed by the U-Net with ResNet50 backbone from [8]. The overall output is a set of keypoints, their descriptors and class

Fig. 3

Overview of the proposed mosaicking framework. The keypoint proposal network (KPN) computes keypoints that are then filtered, according to their semantics, to reject irrelevant keypoints. Registration for mosaicking is performed to register consecutive fetoscopy frames. Warping and blending are performed for visual purposes

Fig. 4

Boxplots of s over n frames (with n in range [1–5]) obtained with (blue) SIFT, (red) [8] and (orange) the proposed framework

Fig. 5

Plot of s with n = 5 computed for all video length. The curves refer to (red) [8] and (orange) the proposed framework

Fig. 6

Mosaics obtained from the six TTTS videos using the method from Bano et al. [8] and the proposed framework. Results refer to the dataset presented in “Dataset” section

Fig. 7

Visual comparison of keypoint computation from (left) SIFT, (middle) E0 and (right) proposed framework