Deep learning driven diagnosis of malignant soft tissue tumors based on dual-modal ultrasound images and clinical indexes

Abstract

Background: Soft tissue tumors (STTs) are benign or malignant superficial neoplasms arising from soft tissues throughout the body with versatile pathological types. Although Ultrasonography (US) is one of the most common imaging tools to diagnose malignant STTs, it still has several drawbacks in STT diagnosis that need improving.

Objectives: The study aims to establish this deep learning (DL) driven Artificial intelligence (AI) system for predicting malignant STTs based on US images and clinical indexes of the patients.

Methods: We retrospectively enrolled 271 malignant and 462 benign masses to build the AI system using 5-fold validation. A prospective dataset of 44 malignant masses and 101 benign masses was used to validate the accuracy of system. A multi-data fusion convolutional neural network, named ultrasound clinical soft tissue tumor net (UC-STTNet), was developed to combine gray scale and color Doppler US images and clinic features for malignant STTs diagnosis. Six radiologists (R1-R6) with three experience levels were invited for reader study.

Results: The AI system achieved an area under receiver operating curve (AUC) value of 0.89 in the retrospective dataset. The diagnostic performance of the AI system was higher than that of one of the senior radiologists (AUC of AI vs R2: 0.89 vs. 0.84, p=0.022) and all of the intermediate and junior radiologists (AUC of AI vs R3, R4, R5, R6: 0.89 vs 0.75, 0.81, 0.80, 0.63; p <0.01). The AI system also achieved an AUC of 0.85 in the prospective dataset. With the assistance of the system, the diagnostic performances and inter-observer agreement of the radiologists was improved (AUC of R3, R5, R6: 0.75 to 0.83, 0.80 to 0.85, 0.63 to 0.69; p<0.01).

Conclusion: The AI system could be a useful tool in diagnosing malignant STTs, and could also help radiologists improve diagnostic performance.

Keywords: artificial intelligence; deep learning; malignant tumor; soft tissue tumor; ultrasound.

Figure 1

Overall study flow of UC-STTNet, the AI system for STTs diagnosis. The AI system was developed on a deep learning frame work using the tumor information from both dual-modal US images, including gray-scale US and color-Doppler US, and clinical features. The AI system could help radiologists in clinical decision-making by providing prediction results of STTs and heatmaps of US images as reference.

Figure 2

Flow chart of the retrospective and prospective patients’ recruitment.

Figure 3

Receiver operating characteristic curves (ROC) of UC-STTNet assessed by 5-fold cross validations and comparing the different level radiologists. 3 (A). ROC of each fold of the AI system and three different levels of radiologists; 3 (B). the average performance of the AI system compared with three levels radiologists.

Figure 4

AUC of the radiologists with and without referring to the AI system. R1 and R2: senior radiologists; R3 and R4: intermediate radiologists; R5 and R6: junior radiologists. For junior radiologist (R5 and R6) and one of the intermediate radiologists (R3), the AUC after the AI assistance were significantly improved.

Figure 5

Examples of the AI system classifying benign and malignant STTs. The AI system diagnosed STTs based on dual-modal US images and clinical features. Heatmaps of the two modalities of US were also provided by the system. The above case is a 28-year-old female with a STTs mass on the subcutaneous layer of the right hand. She had no tumor or surgical history. The tumor was found 12 months ago and had a size of 13×11mm. The AI system diagnosed it as a benign tumor, which was identified as a benign schwannoma by pathology. The other case is a 64-year-old male with a STTs mass on the muscular layer of abdomen. The patient also reported no tumor or surgical history. The tumor was found 10 days ago and had a size of 22×11mm. The AI system diagnosed it as a malignant STTs tumor, which was identified as a metastatic malignant melanoma by pathology.

Figure 6

ROC curves of the AI system on the retrospective and prospective datasets.