Deep learning combining mammography and ultrasound images to predict the malignancy of BI-RADS US 4A lesions in women with dense breasts: a diagnostic study

Abstract

Objectives: The authors aimed to assess the performance of a deep learning (DL) model, based on a combination of ultrasound (US) and mammography (MG) images, for predicting malignancy in breast lesions categorized as Breast Imaging Reporting and Data System (BI-RADS) US 4A in diagnostic patients with dense breasts.

Methods: A total of 992 patients were randomly allocated into the training cohort and the test cohort at a proportion of 4:1. Another, 218 patients were enrolled to form a prospective validation cohort. The DL model was developed by incorporating both US and MG images. The predictive performance of the combined DL model for malignancy was evaluated by sensitivity, specificity, and area under the receiver operating characteristic curve (AUC). The combined DL model was then compared to a clinical nomogram model and to the DL model trained using US image only and to that trained MG image only.

Results: The combined DL model showed satisfactory diagnostic performance for predicting malignancy in breast lesions, with an AUC of 0.940 (95% CI: 0.874-1.000) in the test cohort, and an AUC of 0.906 (95% CI: 0.817-0.995) in the validation cohort, which was significantly higher than the clinical nomogram model, and the DL model for US or MG alone ( P <0.05).

Conclusions: The study developed an objective DL model combining both US and MG imaging features, which was proven to be more accurate for predicting malignancy in the BI-RADS US 4A breast lesions of patients with dense breasts. This model may then be used to more accurately guide clinicians' choices about whether performing biopsies in breast cancer diagnosis.

Figure 1

Flowchart of patient enrollment in the study. In total, 1,210 of the initial 2,925 patients were included in this study, according to the inclusion/exclusion criteria. The included patients were examined by ultrasound (US) and mammography (MG), and had complete clinical information needed for the study. The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

Figure 2

Overall pipeline of the model. The parallel pre-trained ResNet18 model encodes the input images to features. These combined features are then classified by softmax.

Figure 3

Comparison of receiver operating characteristic (ROC) curves between the different models for predicting malignancy in breast lesions of the test (A) and validation (B) cohorts.

Figure 4

Decision curve analysis (DCA) between the different models for predicting malignancy in breast lesions of the test (A) and validation (B) cohorts.

Figure 5

Visualization of two patient examples. Each example shows the corresponding heat map of the patients' US and MG images. Image a shows a 33 years old women's breast lesion and image b shows a 50 years old women's breast lesion, and the red region represents a larger weight, which can be decoded by the color bar on the right. It shows the abilities of our model accurately focusing on the breast lesions. The color of the lesions gets closer to red, the more focusing our model learns on that region.