Transfer learning radiomics based on multimodal ultrasound imaging for staging liver fibrosis

Abstract

Objectives: To propose a transfer learning (TL) radiomics model that efficiently combines the information from gray scale and elastogram ultrasound images for accurate liver fibrosis grading.

Methods: Totally 466 patients undergoing partial hepatectomy were enrolled, including 401 with chronic hepatitis B and 65 without fibrosis pathologically. All patients received elastography and got liver stiffness measurement (LSM) 2-3 days before surgery. We proposed a deep convolutional neural network by TL to analyze images of gray scale modality (GM) and elastogram modality (EM). The TL process was used for liver fibrosis classification by Inception-V3 network which pretrained on ImageNet. The diagnostic performance of TL and non-TL was compared. The value of single modalities, including GM and EM alone, and multimodalities, including GM + LSM and GM + EM, was evaluated and compared with that of LSM and serological indexes. Receiver operating characteristic curve analysis was performed to calculate the optimal area under the curve (AUC) for classifying fibrosis of S4, ≥ S3, and ≥ S2.

Results: TL in GM and EM demonstrated higher diagnostic accuracy than non-TL, with significantly higher AUCs (all p < .01). Single-modal GM and EM both performed better than LSM and serum indexes (all p < .001). Multimodal GM + EM was the most accurate prediction model (AUCs are 0.950, 0.932, and 0.930 for classifying S4, ≥ S3, and ≥ S2, respectively) compared with GM + LSM, GM and EM alone, LSM, and biomarkers (all p < .05).

Conclusions: Liver fibrosis can be staged by a transfer learning modal based on the combination of gray scale and elastogram ultrasound images, with excellent performance.

Key points: • Transfer learning consists in applying to a specific deep learning algorithm that pretrained on another relevant problem, expected to reduce the risk of overfitting due to insufficient medical images. • Liver fibrosis can be staged by transfer learning radiomics with excellent performance. • The most accurate prediction model of transfer learning by Inception-V3 network is the combination of gray scale and elastogram ultrasound images.

Keywords: Deep learning; Elasticity imaging techniques; Hepatitis B; Liver cirrhosis.

Fig. 1

Illustration of the overall transfer learning framework of this study. All the convolutional and pooling layers except the last multinomial logistic classification layer of the Inception-V3 model were taken out as the feature extractor of this study

Fig. 2

Illustration of the 2D SWE measurement and the ROI of transfer learning (TL) in this study. Image of elastogram image (top), gray scale image (bottom), liver stiffness measurement with Q-Box (white circle area), and ROI of TL (red square area)

Fig. 3

Comparison of ROC curves between TL and non-TL for the assessment of liver fibrosis stages in training and test cohort, respectively. a, d S0–S3 versus S4 in training and test cohort. b, e S0–S2 versus S3–S4 (≥ S3) in training and test cohort. c, f S0–S1 versus S2–S4 (≥ S2) in training and test cohort. TL, transfer learning; Non-TL, non-transfer learning

Fig. 4

Comparison of AUCs between GM + EM, GM + LSM, EM, GM, LSM, APRI, and FIB-4 for the assessment of liver fibrosis stages in test cohorts. a S0–S3 versus S4 (S4); b S0–S2 versus S3–S4 (≥ S3); c S0–S1 versus S2–S4 (≥ S2). GM + EM, gray scale modality and elastogram modality; GM + LSM, gray scale modality and liver stiffness measurement; GM, gray scale modality; EM, elastogram modality; LSM, liver stiffness measurement

Fig. 5

The demonstration of elastogram and gray scale modalities of different liver fibrosis stages. a, e Elastogram and gray scale modalities of S0~1. b, f Elastogram and gray scale modalities of S2. c, g Elastogram and gray scale modalities of S3. d, h Elastogram and gray scale modalities of S4