Ensembles of Convolutional Neural Networks and Transformers for Polyp Segmentation

Abstract

In the realm of computer vision, semantic segmentation is the task of recognizing objects in images at the pixel level. This is done by performing a classification of each pixel. The task is complex and requires sophisticated skills and knowledge about the context to identify objects' boundaries. The importance of semantic segmentation in many domains is undisputed. In medical diagnostics, it simplifies the early detection of pathologies, thus mitigating the possible consequences. In this work, we provide a review of the literature on deep ensemble learning models for polyp segmentation and develop new ensembles based on convolutional neural networks and transformers. The development of an effective ensemble entails ensuring diversity between its components. To this end, we combined different models (HarDNet-MSEG, Polyp-PVT, and HSNet) trained with different data augmentation techniques, optimization methods, and learning rates, which we experimentally demonstrate to be useful to form a better ensemble. Most importantly, we introduce a new method to obtain the segmentation mask by averaging intermediate masks after the sigmoid layer. In our extensive experimental evaluation, the average performance of the proposed ensembles over five prominent datasets beat any other solution that we know of. Furthermore, the ensembles also performed better than the state-of-the-art on two of the five datasets, when individually considered, without having been specifically trained for them.

Keywords: computer vision; convolutional neural networks; ensemble; polyp segmentation; transformers.

Figure 1

Two examples of the content of the ETIS-Larib [4] dataset for the semantic segmentation of polyps: (a,c) original images; (b,d) ground truth.

Figure 2

Masks in HSNet. (a) Final segmentation mask. (b–e) Intermediate masks after the sigmoid computation.

Figure 3

An example of images obtained through our data augmentation process. (Top) Original polyp image. (Bottom) Synthetic images created from the original image by data augmentation.

Figure 4

Overall structure of the proposed method, including the training and testing procedures.

Figure 5

Masks in HSNet ensemble with and without smoothing. (a,e) Original image, (b,f) ground truth, (c,g) HSNet ensemble without smoothing, and (d,h) HSNet ensemble with smoothing. False positive pixels are green. False negative pixels are red.