Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Jan 4;14(2):593-607.
doi: 10.1364/BOE.473446. eCollection 2023 Feb 1.

Spatio-temporal classification for polyp diagnosis

Juana González-Bueno Puyal  1   2 , Patrick Brandao  2 , Omer F Ahmad  1 , Kanwal K Bhatia  2 , Daniel Toth  2 , Rawen Kader  1 , Laurence Lovat  1 , Peter Mountney  2 , Danail Stoyanov  1
Affiliations

Spatio-temporal classification for polyp diagnosis

Juana González-Bueno Puyal et al. Biomed Opt Express. .

Abstract

Colonoscopy remains the gold standard investigation for colorectal cancer screening as it offers the opportunity to both detect and resect pre-cancerous polyps. Computer-aided polyp characterisation can determine which polyps need polypectomy and recent deep learning-based approaches have shown promising results as clinical decision support tools. Yet polyp appearance during a procedure can vary, making automatic predictions unstable. In this paper, we investigate the use of spatio-temporal information to improve the performance of lesions classification as adenoma or non-adenoma. Two methods are implemented showing an increase in performance and robustness during extensive experiments both on internal and openly available benchmark datasets.

PubMed Disclaimer

Conflict of interest statement

D.S: Odin Vision Ltd (I, S), D.S: Digital Surgery Ltd (E), L.L: Odin Vision Ltd (I).

Figures

Fig. 1.
Fig. 1.
Examples of polyp appearance variation (with expert polyp boxes in blue) for (a) an adenoma and (d) non-adenoma polyp. The timelines (middle) show example predictions on the adenoma video sequence (b) and non-adenoma sequence (c) - green, red and grey denote correct and incorrect predictions and non-annotated frames, respectively.
Fig. 2.
Fig. 2.
Architectures of the proposed spatio-temporal methods for adenoma/non-adenoma video clip classification.
Fig. 3.
Fig. 3.
Prediction timelines for the same polyp sequence with (a) LRCN, (b) ConvNet averaging and (c) ConvNet - green, red and grey denote correct and incorrect predictions and non-annotated frames, respectively. Note: the spatio-temporal methods present shorter timelines as the last k−1=14 samples (0.6 seconds) did not have enough following frames to create a clip.
Fig. 4.
Fig. 4.
Boxplots showing the per-polyp accuracies for each method.
Fig. 5.
Fig. 5.
Performance for (a) LRCN and (b) ConvNet averaging for different clip cross-correlations - higher cross-correlation implies higher intra-clip similarity and lower variation. 95% confidence intervals are shown with transparency.
Fig. 6.
Fig. 6.
LRCN performance when trained with different clip sizes.
Fig. 7.
Fig. 7.
Classification results examples. The top row shows examples where ConvNet averaging succeeds and LRCN fails, and the bottom row examples where the opposite occurs.
Fig. 8.
Fig. 8.
Models performance based on the quality of the position of the polyp box. The bounding box around each polyp was randomly moved to achieve 9 new boxes with an intersection over union (iou) with the original expert box ranging from 0.05 to 0.95. Area under the curve (auc), accuracy, sensitivity, specificity and per-polyp accuracy are shown. The image on the bottom right shows an example of the position of the original box (red transparency) and boxes obtained with different ious.
See this image and copyright information in PMC

References

    1. Bray F., Ferlay J., Soerjomataram I., Siegel R. L., Torre L. A., Jemal A., “Global cancer statistics 2018: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” CA Cancer J. Clin. 68, 394–424 (2018).10.3322/caac.21492 - DOI - PubMed
    1. Rex D. K., Johnson D. A., Anderson J. C., Schoenfeld P. S., Burke C. A., Inadomi J. M., “American college of gastroenterology guidelines for colorectal cancer screening 2008,” American Journal of Gastroenterology 104(3), 739–750 (2009).10.1038/ajg.2009.104 - DOI - PubMed
    1. Dayyeh B. K. A., Thosani N., Konda V., Wallace M. B., Rex D. K., Chauhan S. S., Hwang J. H., Komanduri S., Manfredi M., Maple J. T., Murad F. M., Siddiqui U. D., Banerjee S., “ASGE technology committee systematic review and meta-analysis assessing the asge pivi thresholds for adopting real-time endoscopic assessment of the histology of diminutive colorectal polyps,” Gastrointestinal Endoscopy 81(2), 455–456 (2015).10.1016/j.gie.2014.12.022 - DOI - PubMed
    1. Mori Y., Kudo S.-E., East J. E., Rastogi A., Bretthauer M., Misawa M., Sekiguchi M., Matsuda T., Saito Y., Ikematsu H., Hotta K., Ohtsuka K., Kudo T., Mori K., “Cost savings in colonoscopy with artificial intelligence-aided polyp diagnosis: an add-on analysis of a clinical trial (with video),” Gastrointestinal Endoscopy 92(4), 905–911.e1 (2020).10.1016/j.gie.2020.03.3759 - DOI - PubMed
    1. Fonollá R., van der Sommen F., Schreuder R. M., Schoon E. J., de With P. H., “Multi-modal classification of polyp malignancy using cnn features with balanced class augmentation,” in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), (IEEE, 2019), pp. 74–78.

LinkOut - more resources