Brain tumor segmentation by combining MultiEncoder UNet with wavelet fusion

Abstract

Background and objective: Accurate segmentation of brain tumors from multimodal magnetic resonance imaging (MRI) holds significant importance in clinical diagnosis and surgical intervention, while current deep learning methods cope with situations of multimodal MRI by an early fusion strategy that implicitly assumes that the modal relationships are linear, which tends to ignore the complementary information between modalities, negatively impacting the model's performance. Meanwhile, long-range relationships between voxels cannot be captured due to the localized character of the convolution procedure.

Method: Aiming at this problem, we propose a multimodal segmentation network based on a late fusion strategy that employs multiple encoders and a decoder for the segmentation of brain tumors. Each encoder is specialized for processing distinct modalities. Notably, our framework includes a feature fusion module based on a 3D discrete wavelet transform aimed at extracting complementary features among the encoders. Additionally, a 3D global context-aware module was introduced to capture the long-range dependencies of tumor voxels at a high level of features. The decoder combines fused and global features to enhance the network's segmentation performance.

Result: Our proposed model is experimented on the publicly available BraTS2018 and BraTS2021 datasets. The experimental results show competitiveness with state-of-the-art methods.

Conclusion: The results demonstrate that our approach applies a novel concept for multimodal fusion within deep neural networks and delivers more accurate and promising brain tumor segmentation, with the potential to assist physicians in diagnosis.

Keywords: 3D discrete wavelet transformer; brain tumor segmentation; multi‐encoder.

FIGURE 1

A sample from the BraTS2021 training dataset is depicted, exhibiting images arranged in the following order: T1, T1ce, T2, FLAIR, and the Ground Truth, which embodies the doctor's manual segmentation annotation. The color codes red, blue, and green correspond to the representation of necrotic tissue, enhancing tumor and edema, respectively.

FIGURE 2

Early fusion and late fusion (early fusion is the combined input of multiple modalities into a network, and late fusion is the separate input of multiple modalities into networks).

FIGURE 3

Overview of our proposed architecture. The whole network has five levels. In the first three layers, only residual blocks (grey) extract features, and in the last two layers, a global context‐aware module (orange) captures global information. The wavelet fusion module (green) fuses the modal features extracted by multiple encoders and concat the fused features to the decoder.

FIGURE 4

The 3D wavelet fusion module.

FIGURE 5

The 3D global context‐aware module.

FIGURE 6

Visual segmentation results of ablation experiments. The non‐enhancing area is red, green indicates edema, and blue represents the enhancing tumor.

FIGURE 7

The segmentation results for several approaches are visualized. Red: non‐enhancing area, Green: edema, Blue: enhancing tumor.

FIGURE 8

Comparison of the convergence speed of different model loss curves at 100 epochs.

FIGURE 9

Boxplots are utilized to facilitate the comparison of various methods concerning both Dice metrics and Hausdorff95 Distance metrics. Dots show outliers.

FIGURE 10

Boxplot and scatter plots of Dice coefficients. (a) and (b) are from the BraTS2018 online validation set, and (c) and (d) are from the BraTS2021 online validation set.