Binary semantic segmentation for detection of prostate adenocarcinoma using an ensemble with attention and residual U-Net architectures

Abstract

An accurate determination of the Gleason Score (GS) or Gleason Pattern (GP) is crucial in the diagnosis of prostate cancer (PCa) because it is one of the criterion used to guide treatment decisions for prognostic-risk groups. However, the manually designation of GP by a pathologist using a microscope is prone to error and subject to significant inter-observer variability. Deep learning has been used to automatically differentiate GP on digitized slides, aiding pathologists and reducing inter-observer variability, especially in the early GP of cancer. This article presents a binary semantic segmentation for the GP of prostate adenocarcinoma. The segmentation separates benign and malignant tissues, with the malignant class consisting of adenocarcinoma GP3 and GP4 tissues annotated from 50 unique digitized whole slide images (WSIs) of prostate needle core biopsy specimens stained with hematoxylin and eosin. The pyramidal digitized WSIs were extracted into image patches with a size of 256 × 256 pixels at a magnification of 20×. An ensemble approach is proposed combining U-Net-based architectures, including traditional U-Net, attention-based U-Net, and residual attention-based U-Net. This work initially considers a PCa tissue analysis using a combination of attention gate units with residual convolution units. The performance evaluation revealed a mean Intersection-over-Union of 0.79 for the two classes, 0.88 for the benign class, and 0.70 for the malignant class. The proposed method was then used to produce pixel-level segmentation maps of PCa adenocarcinoma tissue slides in the testing set. We developed a screening tool to discriminate between benign and malignant prostate tissue in digitized images of needle biopsy samples using an AI approach. We aimed to identify malignant adenocarcinoma tissues from our own collected, annotated, and organized dataset. Our approach returned the performance which was accepted by the pathologists.

Keywords: Adenocarcinoma detection; Attention gate; Binary semantic segmentation; Model-ensemble; Prostate cancer; Residual convolution.

Figure 1. (A) Forward convolutional units (exploited in this work), (B) recurrent convolutional block, (C) residual convolutional units (exploited in this work) and (D) recurrent residual convolutional units (Alom et al., 2018).

Figure 2. Attention U-Net segmentation model and schematic of attention gates (Oktay et al., 2018).

Figure 3. Examples of tissue patterns annotated by the pathologist: (A) normal glands, (B) discrete well-formed glands of GP3, (C) fused glands of GP4, and (D) various patterns in the same picture.

Light blue lines indicate GP3, green represents GP4, and blue corresponds to normal glands.

Figure 4. Preprocessing steps that produced two different datasets of digitized WSI of prostate tissue.

Figure 5. Examples of image patches and their ground truth masks after preprocessing into binary classes (black pixels indicate benign, and white pixels indicate malignant).

Training: (A–D), validation: (E–H), testing: (I–L).

Figure 6. Example images of benign and malignant patches from the training (A and B), validation (C and D), and testing (E and F) sets.

Ground truth masks for benign patches contained only zero values, while those for malignant patches contained only one values.

Figure 7. The proposed binary semantic segmentation of Gleason patterns in prostate adenocarcinoma exploiting an ensemble model approach with attention-based residual U-Net.

Figure 8. Architecture of the proposed ARU-Net model.

Figure 9. Performance evaluations of model ensemble (ME) method and weighted model ensemble (WME) method.

Figure 10. Model ensemble performance evaluations of mean IoU for the validation set (ME, model ensemble; WME, weighted model ensemble).

Figure 11. Ground truth mask of slide 1,346 from the validation set and its corresponding weighted ensemble segmentation map for malignant probability detected by our models.

Figure 12. Ground truth mask of slide 1,929 from the validation set and its corresponding weighted ensemble segmentation map for malignant probability detected by our models.

Figure 13. Early GP segmentation detection for slides 1,923 and 1,942 from the testing set.

Figure 14. Mean-IoU scores of the performance evaluations for pre-trained segmentation models.

Figure 15. Patch amounts of Dataset A after augmentation.