Whole-lesion-aware network based on freehand ultrasound video for breast cancer assessment: a prospective multicenter study

Abstract

Background: The clinical application of artificial intelligence (AI) models based on breast ultrasound static images has been hindered in real-world workflows due to operator-dependence of standardized image acquisition and incomplete view of breast lesions on static images. To better exploit the real-time advantages of ultrasound and more conducive to clinical application, we proposed a whole-lesion-aware network based on freehand ultrasound video (WAUVE) scanning in an arbitrary direction for predicting overall breast cancer risk score.

Methods: The WAUVE was developed using 2912 videos (2912 lesions) of 2771 patients retrospectively collected from May 2020 to August 2022 in two hospitals. We compared the diagnostic performance of WAUVE with static 2D-ResNet50 and dynamic TimeSformer models in the internal validation set. Subsequently, a dataset comprising 190 videos (190 lesions) from 175 patients prospectively collected from December 2022 to April 2023 in two other hospitals, was used as an independent external validation set. A reader study was conducted by four experienced radiologists on the external validation set. We compared the diagnostic performance of WAUVE with the four experienced radiologists and evaluated the auxiliary value of model for radiologists.

Results: The WAUVE demonstrated superior performance compared to the 2D-ResNet50 model, while similar to the TimeSformer model. In the external validation set, WAUVE achieved an area under the receiver operating characteristic curve (AUC) of 0.8998 (95% CI = 0.8529-0.9439), and showed a comparable diagnostic performance to that of four experienced radiologists in terms of sensitivity (97.39% vs. 98.48%, p = 0.36), specificity (49.33% vs. 50.00%, p = 0.92), and accuracy (78.42% vs.79.34%, p = 0.60). With the WAUVE model assistance, the average specificity of four experienced radiologists was improved by 6.67%, and higher consistency was achieved (from 0.807 to 0.838).

Conclusion: The WAUVE based on non-standardized ultrasound scanning demonstrated excellent performance in breast cancer assessment which yielded outcomes similar to those of experienced radiologists, indicating the clinical application of the WAUVE model promising.

Keywords: Artificial intelligence; Breast neoplasms; Deep learning; Diagnosis; Ultrasonography; Video.

Fig. 1

Overview of the retrospective development and prospective external validation dataset workflows. The development dataset was collected in two hospitals before August 2022, while the external validation dataset was collected in two other hospitals after development of the deep learning model. Abbreviations: NCC, National Cancer Center; PUMCH, Peking Union Medical College Hospital

Fig. 2

The schematic architecture of the whole-lesion-aware network for freehand ultrasound video(WAUVE). For a given ultrasound video, (a) the Faster R-CNN with ResNet-34 backbone firstly extracts regions-of-interest (ROIs) from each frame. Then ROIs are cropped to generate video clips for each video, where each adjacent eight frames are grouped as a video clip. Subsequently, (b) Inflated 3D ConvNet (I3D) with ResNet-50 backbone performs malignancy probability prediction for each video clip. Finally, the overall cancer risk score is outputted in the form of the average of the results of all video clips

Fig. 3

Receiver operating characteristic curves for different DL models in the internal validation dataset. Abbreviations: AUC, area under the receiver operating characteristic curve; DL, deep learning

Fig. 4

Diagnostic performance comparison between WAUVE model and four radiologists. Receiver operating characteristic (ROC) curves and points are used to depict the performance of the model and four radiologists, respectively. Additionally, the mean performance of four radiologists is presented by error bars with 95% confidence intervals (CIs), which were calculated based on 9999 bootstraps of the data. Green and purple points represent the performance of radiologists without and with AI assistance, respectively

Fig. 5

Consistency between different radiologists before (a) and after (b) AI assistance. Based on the color bar on the right, the color block darkens with increasing consistency, and the number in the color block area represents the Kappa value between two corresponding radiologists. With AI assistance, the consistency between radiologists was higher than that before AI assistance, and the Kappa values between different radiologists were all improved to different degrees

Fig. 6

Examples of Qualitative illustration and heatmap illustration of our WAUVE model. We present representative frames from the input video and the corresponding heatmaps. The green bounding boxes represent the detected ROI in each frame. For each ROI, the cancer risk score of the video clip (Clip N: prob = X) is presented above the bounding box. In the heatmap, the red regions contribute to high malignancy probability and the blue regions to benign. a The WAUVE model obtained an overall cancer risk score of 0.643, with mostly red-colored regions suggesting a malignant lesion. All four radiologists in the reader study classified this lesion as BI-RADS 4 C, and the final histopathological diagnosis was a malignancy. b The WAUVE model obtained an overall cancer risk score of 0.111, with mostly blue regions suggesting a benign lesion. One radiologist assigned this lesion as BI-RADS 4a and another as 4b, indicating mild to moderate suspicion for malignancy. However, the final diagnosis was a benign fibroadenoma