Computer-aided diagnosis of masses in breast computed tomography imaging: deep learning model with combined handcrafted and convolutional radiomic features

Abstract

Purpose: A computer-aided diagnosis (CADx) system for breast masses is proposed, which incorporates both handcrafted and convolutional radiomic features embedded into a single deep learning model. Approach: The model combines handcrafted and convolutional radiomic signatures into a multi-view architecture, which retrieves three-dimensional (3D) image information by simultaneously processing multiple two-dimensional mass patches extracted along different planes through the 3D mass volume. Each patch is processed by a stream composed of two concatenated parallel branches: a multi-layer perceptron fed with automatically extracted handcrafted radiomic features, and a convolutional neural network, for which discriminant features are learned from the input patches. All streams are then concatenated together into a final architecture, where all network weights are shared and the learning occurs simultaneously for each stream and branch. The CADx system was developed and tested for diagnosis of breast masses ( $N=284$ ) using image datasets acquired with independent dedicated breast computed tomography systems from two different institutions. The diagnostic classification performance of the CADx system was compared against other machine and deep learning architectures adopting handcrafted and convolutional approaches, and three board-certified breast radiologists. Results: On a test set of 82 masses (45 benign, 37 malignant), the proposed CADx system performed better than all other model architectures evaluated, with an increase in the area under the receiver operating characteristics curve (AUC) of $0.05\pm 0.02$ , and achieving a final AUC of 0.947, outperforming the three radiologists ( $\mathrm{AUC}=0.814-0.902$ ). Conclusions: In conclusion, the system demonstrated its potential usefulness in breast cancer diagnosis by improving mass malignancy assessment.

Keywords: breast cancer; breast computed tomography; computer-aided diagnosis; deep learning; radiomics.

Fig. 1

Examples of breast masses and corresponding nine 2D patches (dimensions of $128\times 128\text{pixels}$) from different planar views through the 3D volume. (a) Examples of benign masses and (b) examples of malignant masses. Each column corresponds to the nine views extracted from the same mass.

Fig. 2

U-Net architecture implemented and used for breast mass segmentation on a 2D patch-basis. The numbers above each convolutional block indicate the number of filters used for that convolutional operation.

Fig. 3

Architecture of the CADx system. 2D patches are used as the input to the network and are processed in parallel by a handcrafted branch (U-Net for automatic segmentation, radiomic feature extraction, selection, and an MLP network), and a convolutional branch including a deep CNN. The last FC layers from the two branches are concatenated, further processed by two additional FC layers, and a final logistic unit that outputs the patch malignancy probability. In this double-input architecture, the learning occurs simultaneously for both branches, with the errors backpropagated throughout the whole architecture in an end-to-end fashion.

Fig. 4

(b)–(e) Schematics of the additional model architectures developed and used for comparisons with the proposed (a) CADx system. The CADx system architecture is shown in detail in Fig. 3.

Fig. 5

Schematics of the multi-view output fusion strategies, implemented to merge the 9 predicted probabilities from each view of the mass into a single malignancy score: (a) averaging; (b) majority voting; (c) view malignancy likelihood; (d) multi-view concatenation.

Fig. 6

Examples of (a) benign and (b) malignant mass patches, respective manual ground truth annotations (green contours), and automatic U-Net-based segmentation (red contours). Yellow parts of the contour indicate a perfect overlay between automatic and manual segmentation.

Fig. 7

ROC curve showing the performance of the proposed CADx system and the additional four architectures evaluated for comparison. The five different model architectures were tested on a mass patch-basis ($n=405$ benign patches from 45 masses, and $n=333$ malignant patches from 37 masses).

Fig. 8

Results of the best-performing CADx system (with output fusion through multi-view concatenation) and three board-certified breast radiologists in the classification of the 82 test set breast masses.

Fig. 9

Results of the view malignancy likelihood analysis.