Nuclei Detection and Segmentation of Histopathological Images Using a Feature Pyramidal Network Variant of a Mask R-CNN

Abstract

Cell nuclei interpretation is crucial in pathological diagnostics, especially in tumor specimens. A critical step in computational pathology is to detect and analyze individual nuclear properties using segmentation algorithms. Conventionally, a semantic segmentation network is used, where individual nuclear properties are derived after post-processing a segmentation mask. In this study, we focus on showing that an object-detection-based instance segmentation network, the Mask R-CNN, after integrating it with a Feature Pyramidal Network (FPN), gives mature and reliable results for nuclei detection without the need for additional post-processing. The results were analyzed using the Kumar dataset, a public dataset with over 20,000 nuclei annotations from various organs. The dice score of the baseline Mask R-CNN improved from 76% to 83% after integration with an FPN. This was comparable with the 82.6% dice score achieved by modern semantic-segmentation-based networks. Thus, evidence is provided that an end-to-end trainable detection-based instance segmentation algorithm with minimal post-processing steps can reliably be used for the detection and analysis of individual nuclear properties. This represents a relevant task for research and diagnosis in digital pathology, which can improve the automated analysis of histopathological images.

Keywords: Mask R-CNN; artificial intelligence; digital pathology; histopathology; nuclei detection.

Figure 1

Mask R-CNN architecture showing FPN modification. The red arrows depict the changes in the FPN variant of the Mask R-CNN.

Figure 2

Visualization of the dice score. X: prediction (orange), Y: ground truth (yellow).

Figure 3

Two example analyses of a patch, (left to right). (a) Image, (b) ground truth instances (green bounding box + mask), (c) predicted instances (red bounding box + mask), (d) false positives, (e) false negatives.

Figure 4

Loss convergence plots. (a) The box regression loss, (b) the RPN bounding box loss, (c) the RPN classification loss, (d) the mask binary cross-entropy loss, (e) the classification loss, (f) the total loss.

Figure 5

Three examples of visualizing nuclei. (a) Input image of H&E-stained tissue, (b) ground truth instance, (c) predicted instance using Mask R-CNN, (d) predicted instance using Mask R-CNN + FPN. The yellow arrows show examples of improved detection.