Systematic Analysis of Common Factors Impacting Deep Learning Model Generalizability in Liver Segmentation

Abstract

Purpose: To investigate the effect of training data type on generalizability of deep learning liver segmentation models.

Materials and methods: This Health Insurance Portability and Accountability Act-compliant retrospective study included 860 MRI and CT abdominal scans obtained between February 2013 and March 2018 and 210 volumes from public datasets. Five single-source models were trained on 100 scans each of T1-weighted fat-suppressed portal venous (dynportal), T1-weighted fat-suppressed precontrast (dynpre), proton density opposed-phase (opposed), single-shot fast spin-echo (ssfse), and T1-weighted non-fat-suppressed (t1nfs) sequence types. A sixth multisource (DeepAll) model was trained on 100 scans consisting of 20 randomly selected scans from each of the five source domains. All models were tested against 18 target domains from unseen vendors, MRI types, and modality (CT). The Dice-Sørensen coefficient (DSC) was used to quantify similarity between manual and model segmentations.

Results: Single-source model performance did not degrade significantly against unseen vendor data. Models trained on T1-weighted dynamic data generally performed well on other T1-weighted dynamic data (DSC = 0.848 ± 0.183 [SD]). The opposed model generalized moderately well to all unseen MRI types (DSC = 0.703 ± 0.229). The ssfse model failed to generalize well to any other MRI type (DSC = 0.089 ± 0.153). Dynamic and opposed models generalized moderately well to CT data (DSC = 0.744 ± 0.206), whereas other single-source models performed poorly (DSC = 0.181 ± 0.192). The DeepAll model generalized well across vendor, modality, and MRI type and against externally sourced data.

Conclusion: Domain shift in liver segmentation appears to be tied to variations in soft-tissue contrast and can be effectively bridged with diversification of soft-tissue representation in training data.Keywords: Convolutional Neural Network (CNN), Deep Learning Algorithms, Machine Learning Algorithms, Supervised Learning, CT, MRI, Liver Segmentation Supplemental material is available for this article. © RSNA, 2023.

Keywords: CT; Convolutional Neural Network (CNN); Deep Learning Algorithms; Liver Segmentation; MRI; Machine Learning Algorithms; Supervised Learning.

Figure 1:

Single-source and DeepAll model three-dimensional performance on unseen MRI type, vendor, and modality. Box and whisker plots represent median, first and third quartiles, and minimum and maximum of Dice-Sørensen coefficient (DSC) values for single-source segmentation models trained on (A) T1-weighted precontrast (dynpre), (B) T1-weighted dynamic portal venous phase (dynportal), (C) T1-weighted non–fat-suppressed (t1nfs), (D) proton density opposed-phase (opposed), and (E) T2-weighted single-shot fast spin-echo (ssfse) MR image types and a multisource (F) DeepAll model trained on data pooled from single-source domains. Plots represent testing against source domain test sets (leftmost box) and target domains grouped by unseen MRI sequence (green), vendor (orange), and modality (blue). Unseen MRI sequence type data include holdout data from source domains, as well as T1-weighted dynamic arterial phase (dynarterial), T1-weighted dynamic delayed phase (dyndelay), T1-weighted dynamic hepatobiliary phase (dynhbp), T2-weighted single-shot fast spin-echo with fat suppression (ssfsefs), and T2-weighted fast spin-echo with fat suppression (t2fse). Data from non-Siemens (ns) vendors include the “ns-” prefix. Data from unseen modality include arterial phase CT (ct-arterial), portal phase CT (ct-portal), and delayed phase CT (ct-delay).

Figure 2:

Model three-dimensional performance on unseen MRI types (holdout sets). Heat map represents mean Dice-Sørensen coefficient (DSC) of single-source and DeepAll model performance across MRI types held out from source domains. Source domain data include T1-weighted precontrast (dynpre), T1-weighted dynamic portal venous phase (dynportal), T1-weighted non–fat-suppressed (t1nfs), proton density opposed-phase (opposed), and T2-weighted single-shot fast spin-echo (ssfse). Model results demonstrating performance on holdout data from their respective domains are outlined in blue. P values indicate no statistically significant difference between performance of single-source models tested against intradomain holdout data and DeepAll model performance on that same set.

Figure 3:

Model three-dimensional performance on unseen MRI types. Heat map represents mean Dice-Sørensen coefficient (DSC) of single-source and DeepAll model performance across unseen MRI type domains. Source domain data include T1-weighted precontrast (dynpre), T1-weighted dynamic portal venous phase (dynportal), T1-weighted non–fat-suppressed (t1nfs), proton density opposed-phase (opposed), and T2-weighted single-shot fast spin-echo (ssfse). Target domain data include T1-weighted dynamic arterial phase (dynarterial), T1-weighted dynamic delayed phase (dyndelay), T1-weighted dynamic hepatobiliary phase (dynhbp), T2-weighted single-shot fast spin-echo with fat suppression (ssfsefs), and T2-weighted fast spin-echo with fat suppression (t2fse).

Figure 4:

Model three-dimensional performance on unseen vendor (GE or Philips) domains. Heat map represents mean Dice-Sørensen coefficient (DSC) of single-source and DeepAll model performance across unseen vendor domains. Model results demonstrating performance on cross-vendor data of the same MRI type are outlined in blue. P values indicate no statistically significant difference between performance of single-source models tested against cross-vendor data of the same MRI type and DeepAll model performance on that same set (cross-vendor vs DeepAll). Source domain data include T1-weighted precontrast (dynpre), T1-weighted dynamic portal venous phase (dynportal), T1-weighted non–fat-suppressed (t1nfs), proton density opposed-phase (opposed), and T2-weighted single-shot fast spin-echo (ssfse). Data from non-Siemens (ns) vendors include the “ns-” prefix.

Figure 5:

Model three-dimensional performance on unseen vendor (GE or Philips) and source domains. Heat map represents mean Dice-Sørensen coefficient (DSC) of DeepAll model performance across unseen vendor domains (cross-vendor) and holdout sets from source domain data (intra-domain). P values indicate no statistically significant difference between performance of the DeepAll model tested against cross-vendor data and intradomain holdout sets. Target domain MRI types include T1-weighted precontrast (ns-dynpre), T1-weighted dynamic portal venous phase (ns-dynportal), T1-weighted non–fat-suppressed (ns-t1nfs), proton density opposed-phase (ns-opposed), and T2-weighted single-shot fast spin-echo (ns-ssfse). Data from non-Siemens (ns) vendors include the “ns-” prefix.

Figure 6:

Model three-dimensional performance on unseen modality (CT) domains. Heat map represents mean Dice-Sørensen coefficient (DSC) of single-source and DeepAll model performance across unseen modality domains (CT). Source domain data include T1-weighted precontrast (dynpre), T1-weighted dynamic portal venous phase (dynportal), T1-weighted non–fat-suppressed (t1nfs), proton density opposed-phase (opposed), and T2-weighted single-shot fast spin-echo (ssfse). Data from unseen modality include arterial phase CT (ct-arterial), portal phase CT (ct-portal), and delayed phase CT (ct-delay).