Endomapper dataset of complete calibrated endoscopy procedures

Abstract

Computer-assisted systems are becoming broadly used in medicine. In endoscopy, most research focuses on the automatic detection of polyps or other pathologies, but localization and navigation of the endoscope are completely performed manually by physicians. To broaden this research and bring spatial Artificial Intelligence to endoscopies, data from complete procedures is needed. This paper introduces the Endomapper dataset, the first collection of complete endoscopy sequences acquired during regular medical practice, making secondary use of medical data. Its main purpose is to facilitate the development and evaluation of Visual Simultaneous Localization and Mapping (VSLAM) methods in real endoscopy data. The dataset contains more than 24 hours of video. It is the first endoscopic dataset that includes endoscope calibration as well as the original calibration videos. Meta-data and annotations associated with the dataset vary from the anatomical landmarks, procedure labeling, segmentations, reconstructions, simulated sequences with ground truth and same patient procedures. The software used in this paper is publicly available.

Fig. 1

Overview of the Endomapper Dataset.

Fig. 2

Two examples of calibration images (left, middle). The calibration pattern (right).

Fig. 3

Example of photometric calibration results.

Fig. 4

Illustration of the anatomical regions labeled.

Fig. 5

Directory structure of the dataset.

Fig. 6

Examples for the tool segmentation mask in Seq_009.

Fig. 7

Two clusters from the COLMAP reconstruction after processing Seq_001. For each cluster, it is shown a 3D view of the frames’ poses and colon map points and five RGB images as summary of the cluster frames.

Fig. 8

Comparison of how a regular pixel grid is undistorted by the calibration of each endoscope. Colonoscopes and gastroscopes are separated for easier visualization.

Fig. 9

Distributions of error in the images in prototype calibrations. The line representing the error is not magnified, observe that most of them appear as points as errors are mostly smaller than one pixel.

Fig. 10

Relation between the incoming ray angle θ with the distorted radius r_d. The dotted curves represent the ideal orthogonal and equisolid projection models. The right image is a zoom of the curves to show the small differences between the colonoscope and the gastroscope.

Fig. 11

View angles plotted on top of calibration images from each prototype endoscope. The iso-lines are plotted in 20° intervals.

Fig. 12

Examples of successful retrieval from different sequences of the same patient. The left column contains the queries from the current sequence (Seq_035) while the rest of the columns are the first three retrieved images from the previous sequence (Seq_027).

Fig. 13

ORB-SLAM3 sub-map in Seq_015 between frames 54420 and 55170. The camera undergoes a forward-backwards motion. Right, 3D map in top view, keyframes in blue, map points in red. Left images corresponding to 4 keyframes spread over the trajectory.

Fig. 14

Binary Segmentation examples from Endomapper dataset using different approaches fine-tuned on Endomapper dataset: (a) Original Image (b) Ground-truth manual segmentation (c) MiniNet (d) UNet (e) LinkNet (f) MF-TAPNet.