Localization of Capsule Endoscope in Alimentary Tract by Computer-Aided Analysis of Endoscopic Images

Abstract

Capsule endoscopy is a common method for detecting digestive diseases. The location of a capsule endoscope should be constantly monitored through a visual inspection of the endoscopic images by medical staff to confirm the examination's progress. In this study, we proposed a computer-aided analysis (CADx) method for the localization of a capsule endoscope. At first, a classifier based on a Swin Transformer was proposed to classify each frame of the capsule endoscopy videos into images of the stomach, small intestine, and large intestine, respectively. Then, a K-means algorithm was used to correct outliers in the classification results. Finally, a localization algorithm was proposed to determine the position of the capsule endoscope in the alimentary tract. The proposed method was developed and validated using videos of 204 consecutive cases. The proposed CADx, based on a Swin Transformer, showed a precision of 93.46%, 97.28%, and 98.68% for the classification of endoscopic images recorded in the stomach, small intestine, and large intestine, respectively. Compared with the landmarks identified by endoscopists, the proposed method demonstrated an average transition time error of 16.2 s to locate the intersection of the stomach and small intestine, as well as 13.5 s to locate that of the small intestine and the large intestine, based on the 20 validation videos with an average length of 3261.8 s. The proposed method accurately localizes the capsule endoscope in the alimentary tract and may replace the laborious real-time visual inspection in capsule endoscopic examinations.

Keywords: capsule endoscopy; computer-aided analysis; deep learning; transformer.

Figure 1

Frames from three capsule endoscopy videos. The intersections between the stomach and small intestine and between the small intestine and colon are indicated by a red dashed line.

Figure 2

Flowchart of this study including data pre–processing, dataset division, and intersection localization.

Figure 3

Visualization of the results of $CPV$ and K-means algorithm in different gastrointestinal regions. (a,b) use colors to represent the stomach (brown), small intestine (orange), and large intestine (blue). (c) uses stacked bars to show $CP$ values for each region per frame. (d,e) represent the $CPV$ using a color gradient, where each square represents a frame and its color intensity correlates with the $CPV$ magnitude, indicating the confidence of the model and the prediction of the predominant class.

Figure 4

Example of our proposed localization algorithm. (a) New composite predict value for each processed frame, obtained from the $CPV$ after K-means clustering. (b) First waiting area (WA(1)) and waiting area size. (c) Second waiting area (WA(2)) and waiting area slide size. (d) Final waiting area (WA(Final)) of the video. (e) Waiting area average value (WAAV) for all WAs in a video. WAAV is the key indicator for detecting the intersection of the stomach–small intestine and small intestine–large intestine. (f) Intersection of stomach and small intestine (ISS) and intersection of small intestine and large intestine (ISL). Arrows indicate the ISS and ISL.

Figure 5

Examples of misclassified frames from capsule endoscopy videos. (a) shows a stomach frame incorrectly classified as the small intestine; (b) shows a stomach frame misclassified as the large intestine; (c) shows a small intestine frame misclassified as the stomach; (d) shows a small intestine frame misclassified as the large intestine; (e) shows a large intestine frame misclassified as the stomach; and (f) shows a large intestine frame misclassified as the small intestine.

Figure 6

Comparison of the results of localization of organ intersection points in capsule endoscopy videos using different methods. (a) Expert annotations from an experienced endoscopist; (b) Results from the Swin Transformer model; (c) Results combining the Swin Transformer model with the K-means algorithm; (d) Results from combining the Swin Transformer model with the OPLA algorithm; (e) Results combining the Swin Transformer model with both the K-means and OPLA algorithms. Red, yellow, and blue represent the stomach, small intestine, and large intestine, respectively. Yellow and blue arrows indicate the intersections between stomach and small intestine and small intestine and large intestine intersections, respectively. The gradient color bar at the bottom represents the continuous spectrum of organ Swin Transformer, with values ranging from 1 (stomach) to 3 (large intestine).