Deep learning-based predictive model for pathological complete response to neoadjuvant chemotherapy in breast cancer from biopsy pathological images: a multicenter study

Abstract

Introduction: Early predictive pathological complete response (pCR) is beneficial for optimizing neoadjuvant chemotherapy (NAC) strategies for breast cancer. The hematoxylin and eosin (HE)-stained slices of biopsy tissues contain a large amount of information on tumor epithelial cells and stromal. The fusion of pathological image features and clinicopathological features is expected to build a model to predict pCR of NAC in breast cancer. Methods: We retrospectively collected a total of 440 breast cancer patients from three hospitals who underwent NAC. HE-stained slices of biopsy tissues were scanned to form whole-slide images (WSIs), and pathological images of representative regions of interest (ROI) of each WSI were selected at different magnifications. Based on several different deep learning models, we propose a novel feature extraction method on pathological images with different magnifications. Further, fused with clinicopathological features, a multimodal breast cancer NAC pCR prediction model based on a support vector machine (SVM) classifier was developed and validated with two additional validation cohorts (VCs). Results: Through experimental validation of several different deep learning models, we found that the breast cancer pCR prediction model based on the SVM classifier, which uses the VGG16 model for feature extraction of pathological images at ×20 magnification, has the best prediction efficacy. The area under the curve (AUC) of deep learning pathological model (DPM) were 0.79, 0.73, and 0.71 for TC, VC1, and VC2, respectively, all of which exceeded 0.70. The AUCs of clinical model (CM), a clinical prediction model established by using clinicopathological features, were 0.79 for TC, 0.73 for VC1, and 0.71 for VC2, respectively. The multimodal deep learning clinicopathological model (DPCM) established by fusing pathological images and clinicopathological features improved the AUC of TC from 0.79 to 0.84. The AUC of VC2 improved from 0.71 to 0.78. Conclusion: Our study reveals that pathological images of HE-stained slices of pre-NAC biopsy tissues can be used to build a pCR prediction model. Combining pathological images and clinicopathological features can further enhance the predictive efficacy of the model.

Keywords: breast cancer; deep learning; neoadjuvant chemotherapy; pathological complete response; pathological images.

FIGURE 1

Flowchart of patient enrollment. A total of 440 patients with WSI were enrolled from three hospitals.

FIGURE 2

The implementation framework structure of pCR prediction model. (A). Transfer learning based feature extraction for pathological images (TLFEPI Module): ROIs selected from each WSI under different magnifications were feature extracted using transfer learning. (B). Clinicopathological feature extraction (CPFE Module): The clinicopathological features were analyzed by univariate analysis and logistics regression analysis. (C). SVM-based multimodal feature pCR prediction model (SMFPM Module): Feature fusion of pathological images and clinicopathology features, using SVM support vector machine classification to construct a multimodal feature pCR prediction model. pCR: pathologic complete response.

FIGURE 3

Representative ROIs in different magnifications. (A) ×.4 magnification (B) ×.10 magnification (C) ×.20 magnification (D) ×.40 magnification.

FIGURE 4

Breast cancer pCR prediction model based on multimodal features and SVM classifier. The VGG16 deep learning model was used to extract features from 20X pathological images. The weights in the VGG16 model, which has been trained on the ImageNet dataset, are transferred into the 13-layer convolutional layer of the feature extraction model. Fine-tune the parameters of the fully-connected layer based on the pCR and non-pCR data. After completing the fine-tuned training, the 13-layer convolutional layer was used as a feature extraction network to obtain a 512-dimensional pathology slice image feature vector. Dimensionality reduction is achieved by a fully connected layer with 7 channels. Then the pathological image features and clinicopathology text features are fused into multimodal features, which were inputted into a radial basis function based support vector machine (SVM) for pCR prediction.

FIGURE 5

The ROC curve for pCR prediction performance in the (A) DPM, (B) CM, and (C) DPCM among all cohorts. AUC: area under the receiver operating characteristic. CM: clinical model. DPCM: deep learning clinicopathological model. DPM: deep learning pathological model. ROC: receiver operating characteristic.

FIGURE 6

The ROC curve for pCR prediction performance of DPM and DPCM in different molecular subtypes in validation cohorts. The ROC curve for pCR prediction performance of DPM in (A). HR+ and Her2-, (B). Her2+, (C). TNBC. The ROC curve for pCR prediction performance of DPCM in (D). HR+ and Her2-, (E). Her2+, (F). TNBC. ROC: receiver operating characteristic. AUC: area under the receiver operating characteristic. DPM: deep learning pathological model. DPCM: deep learning clinicopathological model. HR: hormone receptor. Her2: human epidermal growth factor receptor 2. TNBC: triple-negative breast cancer.