Deep Multi-Magnification Networks for multi-class breast cancer image segmentation

Abstract

Pathologic analysis of surgical excision specimens for breast carcinoma is important to evaluate the completeness of surgical excision and has implications for future treatment. This analysis is performed manually by pathologists reviewing histologic slides prepared from formalin-fixed tissue. In this paper, we present Deep Multi-Magnification Network trained by partial annotation for automated multi-class tissue segmentation by a set of patches from multiple magnifications in digitized whole slide images. Our proposed architecture with multi-encoder, multi-decoder, and multi-concatenation outperforms other single and multi-magnification-based architectures by achieving the highest mean intersection-over-union, and can be used to facilitate pathologists' assessments of breast cancer.

Keywords: Breast cancer; Computational pathology; Deep Multi-Magnification Network; Multi-class image segmentation; Partial annotation.

Figure 1:

Introduction of a Deep Single-Magnification Network (DSMN) and a Deep Multi-Magnification Network (DMMN) for tissue segmentation of whole slide images. (a) A DSMN looks at a patch from a single magnification from a whole slide image with limited field-of-view to generate the corresponding multi-class tissue segmentation prediction. (b) A DMMN looks at a set of patches from multiple magnifications from a whole slide image to have wider field-of-view to generate the corresponding multi-class tissue segmentation prediction. The DMMN can learn both cellular features from a higher magnification and architectural growth patterns from a lower magnification. Here, carcinoma is predicted in red, benign epithelial in blue, background in yellow, stroma in green, necrotic in gray, and adipose in orange.

Figure 2:

Block diagram of the proposed method with our Deep Multi-Magnification Network. The first step of our method is to partially annotate training whole slide images. After extracting training patches from the partial annotations and balancing the number of pixels between classes, our Deep Multi-Magnification Network is trained. The trained network is used for multi-class tissue segmentation of whole slide images.

Figure 3:

An example of partial annotation. (a) A whole slide image from breast tissue. (b) A partially annotated image where multiple tissue subtypes are annotated in distinct colors and white regions are unlabeled. (c) The partial annotation overlaid on the whole slide image. Subtype components are annotated without cropping while reducing the thickness of unlabeled regions between the subtype components. Here, carcinoma is annotated in red, benign epithelial in blue, background in yellow, stroma in green, necrotic in gray, and adipose in orange.

Figure 4:

CNN architectures for multi-class tissue segmentation of a Deep Single-Magnification Network (DSMN) in (a) utilizing a patch from a single magnification and Deep Multi-Magnification Networks (DMMNs) in (b-e) utilizing multiple patches in various magnifications. (a) U-Net [17] is used as our DSMN architecture. (b) Single-Encoder Single-Decoder (DMMN-S2) is a DMMN architecture where multiple patches are concatenated and used as an input to the U-Net architecture. (c) Multi-Encoder Single-Decoder (DMMN-MS) is a DMMN architecture having only one decoder. (d) Multi-Encoder Multi-Decoder Single-Concatenation (DMMN-M2S) is a DMMN architecture where feature maps from multiple magnifications are only concatenated at the final layer. (e) Our proposed Multi-Encoder Multi-Decoder Multi-Concatenation (DMMN-M3) is a DMMN architecture where feature maps are concatenated during intermediate layers to enrich feature maps in the decoder of the highest magnification.

Figure 5:

Class balancing using elastic deformation in the training breast dataset.

Figure 6:

Multi-class tissue segmentation predictions of a whole slide image from Dataset-I using two Deep Single-Magnification Networks (DSMNs), SegNet [16] and U-Net [17], and four Deep Multi-Magnification Networks (DMMNs), Single-Encoder Single-Decoder (DMMN-S2), Multi-Encoder Single-Decoder (DMMN-MS), Multi-Encoder Multi-Decoder Single-Concatenation (DMMN-M2S), and our proposed Multi-Encoder Multi-Decoder Multi-Concatenation (DMMN-M3).

Figure 7:

Multi-class tissue segmentation predictions of invasive ductal carcinoma (IDC) in red from Dataset-I using two Deep Single-Magnification Networks (DSMNs), SegNet [16] and U-Net [17], and four Deep Multi-Magnification Networks (DMMNs), Single-Encoder Single-Decoder (DMMN-S2), Multi-Encoder Single-Decoder (DMMN-MS), Multi-Encoder Multi-Decoder Single-Concatenation (DMMN-M2S), and our proposed Multi-Encoder Multi-Decoder Multi-Concatenation (DMMN-M3).

Figure 8:

Multi-class tissue segmentation predictions of a whole slide image from Dataset-I using two Deep Single-Magnification Networks (DSMNs), SegNet [16] and U-Net [17], and four Deep Multi-Magnification Networks (DMMNs), Single-Encoder Single-Decoder (DMMN-S2), Multi-Encoder Single-Decoder (DMMN-MS), Multi-Encoder Multi-Decoder Single-Concatenation (DMMN-M2S), and our proposed Multi-Encoder Multi-Decoder Multi-Concatenation (DMMN-M3).

Figure 9:

Multi-class tissue segmentation predictions of benign epithelial in blue from Dataset-I using two Deep Single-Magnification Networks (DSMNs), SegNet [16] and U-Net [17], and four Deep Multi-Magnification Networks (DMMNs), Single-Encoder Single-Decoder (DMMN-S2), Multi-Encoder Single-Decoder (DMMN-MS), Multi-Encoder Multi-Decoder Single-Concatenation (DMMN-M2S), and our proposed Multi-Encoder Multi-Decoder Multi-Concatenation (DMMN-M3).

Figure 10:

Multi-class tissue segmentation predictions of a whole slide image from Dataset-II using two Deep Single-Magnification Networks (DSMNs), SegNet [16] and U-Net [17], and four Deep Multi-Magnification Networks (DMMNs), Single-Encoder Single-Decoder (DMMN-S2), Multi-Encoder Single-Decoder (DMMN-MS), Multi-Encoder Multi-Decoder Single-Concatenation (DMMN-M2S), and our proposed Multi-Encoder Multi-Decoder Multi-Concatenation (DMMN-M3).

Figure 11:

Multi-class tissue segmentation predictions of ductal carcinoma in situ (DCIS) in red from Dataset-II using two Deep Single-Magnification Networks (DSMNs), SegNet [16] and U-Net [17], and four Deep Multi-Magnification Networks (DMMNs), Single-Encoder Single-Decoder (DMMN-S2), Multi-Encoder Single-Decoder (DMMN-MS), Multi-Encoder Multi-Decoder Single-Concatenation (DMMN-M2S), and our proposed Multi-Encoder Multi-Decoder Multi-Concatenation (DMMN-M3).

Figure 12:

Confusion matrices evaluating carcinoma, benign epithelial, stroma, necrotic, adipose, and background segmentation on Dataset-I based on two Deep Single-Magnification Networks (DSMNs), SegNet [16] and U-Net [17], and four Deep Multi-Magnification Networks (DMMNs), Single-Encoder Single-Decoder (DMMN-S2), Multi-Encoder Single-Decoder (DMMN-MS), Multi-Encoder Multi-Decoder Single-Concatenation (DMMN-M2S), and our proposed Multi-Encoder Multi-Decoder Multi-Concatenation (DMMN-M3).

Figure 13:

Confusion matrices evaluating carcinoma, benign epithelial, and stroma segmentation on Dataset-II based on two Deep Single-Magnification Networks (DSMNs), SegNet [16] and U-Net [17], and four Deep Multi-Magnification Networks (DMMNs), Single-Encoder Single-Decoder (DMMN-S2), Multi-Encoder Single-Decoder (DMMN-MS), Multi-Encoder Multi-Decoder Single-Concatenation (DMMN-M2S), and our proposed Multi-Encoder Multi-Decoder Multi-Concatenation (DMMN-M3). Necrotic, adipose, and background are excluded from the confusion matrices on Dataset-II due to the lack of pixels being evaluated.

Figure 14:

Multi-class tissue segmentation predictions of well-differentiated carcinomas in red from Dataset-II using our proposed Multi-Encoder Multi-Decoder Multi-Concatenation (DMMN-M3).