Region Segmentation of Whole-Slide Images for Analyzing Histological Differentiation of Prostate Adenocarcinoma Using Ensemble EfficientNetB2 U-Net with Transfer Learning Mechanism

Abstract

Recent advances in computer-aided detection via deep learning (DL) now allow for prostate cancer to be detected automatically and recognized with extremely high accuracy, much like other medical diagnoses and prognoses. However, researchers are still limited by the Gleason scoring system. The histopathological analysis involved in assigning the appropriate score is a rigorous, time-consuming manual process that is constrained by the quality of the material and the pathologist's level of expertise. In this research, we implemented a DL model using transfer learning on a set of histopathological images to segment cancerous and noncancerous areas in whole-slide images (WSIs). In this approach, the proposed Ensemble U-net model was applied for the segmentation of stroma, cancerous, and benign areas. The WSI dataset of prostate cancer was collected from the Kaggle repository, which is publicly available online. A total of 1000 WSIs were used for region segmentation. From this, 8100 patch images were used for training, and 900 for testing. The proposed model demonstrated an average dice coefficient (DC), intersection over union (IoU), and Hausdorff distance of 0.891, 0.811, and 15.9, respectively, on the test set, with corresponding masks of patch images. The manipulation of the proposed segmentation model improves the ability of the pathologist to predict disease outcomes, thus enhancing treatment efficacy by isolating the cancerous regions in WSIs.

Keywords: U-Net; deep learning; histological; prostate adenocarcinoma; segmentation; transfer learning.

Figure 1

Example of patch images cropped from WSIs. (a–c) Original samples. (d–f) Corresponding ground-truth samples.

Figure 2

The entire process of region segmentation of WSIs for diagnosis of Prostate Adenocarcinoma. (a) The entire dataset of whole-slide images and ground-truth samples. (b) Patch images for training. (c) Patch images for testing. (d) Classification models for training and testing. (e) WSI prediction and auto annotation of stroma, benign, and cancer regions.

Figure 3

Examples of image preprocessing. (a) Original WSI. (b) Threshold result of (a). (c) Pathologist-annotated/ground-truth image with several nonoverlapping blocks for patching. The violet and yellow colors represent tissue regions with Gleason scores of 4 and 5, respectively. (d) The result of patching from the original WSI.

Figure 4

Approaches for training pretrained models based on transfer learning. The weights of pretrained layers in the base models are modified during the learning process, whereas the frozen layers retain their weights (i.e., not modified). (a) A basic transfer learning approach, where the pretrained model is loaded and trained using the extracted features. (b–d) The proposed transfer learning approach, where three freezing techniques are applied and features from each model are concatenated to perform training.

Figure 5

The architecture of UNet and the pretrained convolution neural network (i.e., backbones for feature extraction). The parts of the backbone network marked with red, blue, green, and yellow signify Resnet34-UNet, ResNeXt50-UNet, InceptionV3-UNet, and EfficientNetB2-UNet.

Figure 6

Implementation of the proposed ensemble-based segmentation model. Conv: convolution; MB Conv: mobile inverted bottleneck convolution.

Figure 7

Comparison of segmentation examples from the proposed model with existing methods. Red, blue, and green colors signify stroma, benign, and cancer regions, respectively. The size of each tile in test set is the same as the train dataset, which is 256 × 256 pixels at 0.5$\mathsf{\mu }\mathrm{m}$/pixel.

Figure 8

Slide-level prediction (stroma, benign, and cancer). (a) Original WSI. (b) Annotated WSI. (c) Predicted WSI. The slide-level visualization is carried out at 10× magnification.