Transductive Transfer Learning for Domain Adaptation in Brain Magnetic Resonance Image Segmentation

Abstract

Segmentation of brain images from Magnetic Resonance Images (MRI) is an indispensable step in clinical practice. Morphological changes of sub-cortical brain structures and quantification of brain lesions are considered biomarkers of neurological and neurodegenerative disorders and used for diagnosis, treatment planning, and monitoring disease progression. In recent years, deep learning methods showed an outstanding performance in medical image segmentation. However, these methods suffer from generalisability problem due to inter-centre and inter-scanner variabilities of the MRI images. The main objective of the study is to develop an automated deep learning segmentation approach that is accurate and robust to the variabilities in scanner and acquisition protocols. In this paper, we propose a transductive transfer learning approach for domain adaptation to reduce the domain-shift effect in brain MRI segmentation. The transductive scenario assumes that there are sets of images from two different domains: (1) source-images with manually annotated labels; and (2) target-images without expert annotations. Then, the network is jointly optimised integrating both source and target images into the transductive training process to segment the regions of interest and to minimise the domain-shift effect. We proposed to use a histogram loss in the feature level to carry out the latter optimisation problem. In order to demonstrate the benefit of the proposed approach, the method has been tested in two different brain MRI image segmentation problems using multi-centre and multi-scanner databases for: (1) sub-cortical brain structure segmentation; and (2) white matter hyperintensities segmentation. The experiments showed that the segmentation performance of a pre-trained model could be significantly improved by up to 10%. For the first segmentation problem it was possible to achieve a maximum improvement from 0.680 to 0.799 in average Dice Similarity Coefficient (DSC) metric and for the second problem the average DSC improved from 0.504 to 0.602. Moreover, the improvements after domain adaptation were on par or showed better performance compared to the commonly used traditional unsupervised segmentation methods (FIRST and LST), also achieving faster execution time. Taking this into account, this work presents one more step toward the practical implementation of deep learning algorithms into the clinical routine.

Keywords: brain; deep learning; domain adaptation; magnetic resonance imaging; segmentation; sub-cortical structures; transductive learning; white matter hyperintensities.

Figure 1

The CNN architecture has three convolutional branches and a branch for spatial priors. 2D patches of size 32 × 32 pixels are extracted from three orthogonal views of a 3D volume. For sub-cortical structure segmentation, the spatial prior branch accepts a vector of size 15 with atlas probabilities for each of the 14 structures plus the background, whereas for the WMH lesion segmentation the vector size is three corresponding to white matter, gray matter and cerebrospinal fluid. Histogram loss is computed from the activation maps of the layers, highlighted with dashed blue rectangles. (A) CNN pipeline; (B) Convolutional layers.

Figure 2

Illustration of some activation maps for (A) source, (B) source after applying histogram matching to the target, and (C) target. Here, 36 example activation maps from the third convolutional layer are shown with the “seismic” color-map to visually emphasise the differences in magnitudes of the activation maps.

Figure 3

Transductive domain adaptation training pipeline using histogram loss. A_i and B_i are feature maps extracted from the ith layers of the CNN. L is the number of layers on which the histogram loss is computed. Segmentation loss, in our case cross-entropy loss, is computed using the source ground truth (GT) labels. LC (LogCosh)—logarithm of hyperbolic cosine function.

Figure 4

Comparison of sub-cortical structure segmentation between direct testing (Baseline) and after domain adaptation (TDA). Black dots refer to each subject volume in the target dataset. The connecting lines show correspondence for improved (green) and decreased (red) DSC values.

Figure 5

Qualitative results for sub-cortical structure segmentation: (A) Ground truth; (B) FIRST segmentation; (C) Pre-trained baseline CNN output without domain adaptation; (D) After domain adaptation. Arrows indicate: top → pallidum; bottom → thalamus.

Figure 6

Comparison of WMH lesion segmentation between direct testing (Baseline) and after domain adaptation (TDA). Black dots refer to each subject volume in the target dataset. The connecting lines show correspondence for improved (green) and decreased (red) DSC values.

Figure 7

Qualitative results for WMH lesion segmentation. Small lesion (A) and large lesion (B) segmentation improvements are shown. The bottom row depicts zoomed regions of interests shown in blue rectangles on whole-brain images (top row).

Figure 8

Inter-operator variability in the lesion ground truth masks for the: (A,B) WMH 2017; and (C,D) VH datasets. Blue ellipses indicate the hyperintense tissues near the ventricles.