Toward a navigation framework for fetoscopy

Abstract

Purpose: Fetoscopic laser photocoagulation of placental anastomoses is the most effective treatment for twin-to-twin transfusion syndrome (TTTS). A robust mosaic of placenta and its vascular network could support surgeons' exploration of the placenta by enlarging the fetoscope field-of-view. In this work, we propose a learning-based framework for field-of-view expansion from intra-operative video frames.

Methods: While current state of the art for fetoscopic mosaicking builds upon the registration of anatomical landmarks which may not always be visible, our framework relies on learning-based features and keypoints, as well as robust transformer-based image-feature matching, without requiring any anatomical priors. We further address the problem of occlusion recovery and frame relocalization, relying on the computed features and their descriptors.

Results: Experiments were conducted on 10 in-vivo TTTS videos from two different fetal surgery centers. The proposed framework was compared with several state-of-the-art approaches, achieving higher [Formula: see text] on 7 out of 10 videos and a success rate of [Formula: see text] in occlusion recovery.

Conclusion: This work introduces a learning-based framework for placental mosaicking with occlusion recovery from intra-operative videos using a keypoint-based strategy and features. The proposed framework can compute the placental panorama and recover even in case of camera tracking loss where other methods fail. The results suggest that the proposed framework has large potential to pave the way to creating a surgical navigation system for TTTS by providing robust field-of-view expansion.

Keywords: Fetal surgery; Fetoscopy; Mosaicking; Occlusion recovery; Twin-to-twin transfusion syndrome.

Fig. 1

Overview of the proposed framework for fetoscopic images mosaicking. During the feature extraction phase, features from different pyramid levels are extracted from the input frames (A, B) and transformed by self-attention and cross-attention ($F_c^A$ and $F_c^B$). $F_f^A$ and $F_f^B$ indicate fine features, while $F_c^A$ and $F_c^B$ coarse features. A matching module performs coarse matches and a further refinement, producing keypoints (${\mathrm{Kpts}}^{\mathrm{A}}$ and ${\mathrm{Kpts}}^{\mathrm{B}}$) and descriptors (${\mathrm{Descr}}^{\mathrm{A}}$ and ${\mathrm{Descr}}^{\mathrm{B}}$). Descriptors are used for keyframe extraction, keypoints for homography estimation and then panorama reconstruction. Coarse features are necessary to perform recovery on the global panorama through a comparison with the keyframes found

Fig. 2

Graphical overview of the keyframe extraction algorithm, described in Sect. "Keyframes extraction". Keyframes are highlighted in red (i.e., ${\text{KF}}_0,{\text{KF}}_1$), while the small colored dots are the keypoints, the arrows highlight keypoints matched between frames

Fig. 3

Performance comparison in terms of ${\text{SSIM}}_5$ between (in order from left to right) Bano et al. [9] (EM1), SIFT (EM2), ORB (EM3) and the proposed method (EM4). Wilcoxon statistical tests have been performed to assess statistical differences (${}^{\ast }p<0.05{,}^{\ast \ast }p<0.01{,}^{\ast \ast \ast }p<0.001$)

Fig. 4

Mosaicking comparison between Bano et al. (EM1), SIFT (EM2), ORB (EM3) and the proposed method (EM4) on four dataset videos. Outcomes show a large variability between different methods and between different videos