Fusion of whole and part features for the classification of histopathological image of breast tissue

Abstract

Purpose: Nowadays Computer-Aided Diagnosis (CAD) models, particularly those based on deep learning, have been widely used to analyze histopathological images in breast cancer diagnosis. However, due to the limited availability of such images, it is always tedious to train deep learning models that require a huge amount of training data. In this paper, we propose a new deep learning-based CAD framework that can work with less amount of training data.

Methods: We use pre-trained models to extract image features that can then be used with any classifier. Our proposed features are extracted by the fusion of two different types of features (foreground and background) at two levels (whole-level and part-level). Foreground and background feature to capture information about different structures and their layout in microscopic images of breast tissues. Similarly, part-level and whole-level features capture are useful in detecting interesting regions scattered in high-resolution histopathological images at local and whole image levels. At each level, we use VGG16 models pre-trained on ImageNet and Places datasets to extract foreground and background features, respectively. All features are extracted from mid-level pooling layers of such models.

Results: We show that proposed fused features with a Support Vector Classifier (SVM) produce better classification accuracy than recent methods on BACH dataset and our approach is orders of magnitude faster than the best performing recent method (EMS-Net).

Conclusion: We believe that our method would be another alternative in the diagnosis of breast cancer because of performance and prediction time.

Keywords: Breast cancer; Computer-aided diagnosis; Deep learning; Histology; Histopathological images; Image classification.

Fig. 1

Grad-CAM visualization of five pooling layers of VGG16 for the input image at whole-level and part-level using both foreground and background features, where $p_1$ to $p_5$ represent the five pooling layers of VGG16 models

Fig. 2

Sampled example H&E images from four classes of the BACH dataset: a normal, b benign, c in situ, and d invasive

Fig. 3

Diagram showing the features map of the corresponding input image at whole-level and part-level using both foreground and background features, where $p_1$ to $p_5$ represent the five pooling layers of VGG16 models

Fig. 4

Block diagram of the proposed method, where $F{W}_j(I)$ and $F{P}_j(I)$ represents the foreground features extracted from jth pooling layers ($j\in \{3,4\}$) at whole-level and part-level, respectively. Similarly, $B{W}_j(I)$ and $B{P}_j(I)$ represents the background features extracted from the jth pooling layer at whole-level and part-level, respectively

Fig. 5

Patch-level features extraction for the proposed method. Note that the diagram utlizes the jth pooling layer to extract the aggregated foreground ($(F{P}_j(I)$) and background features ($B{P}_j(I)$) using both foreground (F) and background (B) information

Fig. 6

Confusion matrix of our method on the testing split of a Set 1, b Set 2, c Set 3, d Set 4, and e Set 5