Clinical value of radiomics and machine learning in breast ultrasound: a multicenter study for differential diagnosis of benign and malignant lesions

Abstract

Objectives: We aimed to assess the performance of radiomics and machine learning (ML) for classification of non-cystic benign and malignant breast lesions on ultrasound images, compare ML's accuracy with that of a breast radiologist, and verify if the radiologist's performance is improved by using ML.

Methods: Our retrospective study included patients from two institutions. A total of 135 lesions from Institution 1 were used to train and test the ML model with cross-validation. Radiomic features were extracted from manually annotated images and underwent a multistep feature selection process. Not reproducible, low variance, and highly intercorrelated features were removed from the dataset. Then, 66 lesions from Institution 2 were used as an external test set for ML and to assess the performance of a radiologist without and with the aid of ML, using McNemar's test.

Results: After feature selection, 10 of the 520 features extracted were employed to train a random forest algorithm. Its accuracy in the training set was 82% (standard deviation, SD, ± 6%), with an AUC of 0.90 (SD ± 0.06), while the performance on the test set was 82% (95% confidence intervals (CI) = 70-90%) with an AUC of 0.82 (95% CI = 0.70-0.93). It resulted in being significantly better than the baseline reference (p = 0.0098), but not different from the radiologist (79.4%, p = 0.815). The radiologist's performance improved when using ML (80.2%), but not significantly (p = 0.508).

Conclusions: A radiomic analysis combined with ML showed promising results to differentiate benign from malignant breast lesions on ultrasound images.

Key points: • Machine learning showed good accuracy in discriminating benign from malignant breast lesions • The machine learning classifier's performance was comparable to that of a breast radiologist • The radiologist's accuracy improved with machine learning, but not significantly.

Keywords: Breast cancer; Machine learning; Ultrasound.

Fig. 1

Examples of lesion annotation. The upper row (a, b) shows placement of a region of interest on a benign lesion, while c and d depict a malignant lesion before and after manual segmentation.

Fig. 2

Flowchart of the patient selection process. Pts, patients; BLs, breast lesions

Fig. 3

Receiver operating characteristic curve of the machine learning classifier for distinguishing benign and malignant lesions in the test set

Fig. 4

Calibration curve plot of the model in the test set. Average predicted probability is represented in the x-axis while the proportion of malignant lesions in the y-axis

Fig. 5

B-mode US images of a benign (a) and malignant (b) breast lesion initially misclassified by the expert radiologist and correctly diagnosed with the availability of ML reading. a A case of a 13-year-old patient with a 4-cm oval breast lesion with circumscribed margins but heterogeneous echo-pattern, proved to be a sclerosing papilloma after surgical excision. b A case of a 59-year-old patient with a 5-mm oval, hypoechoic breast lesion with circumscribed margins, histologically proved as Luminal A, G1, ductal invasive carcinoma