Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Nov;25(11):e14527.
doi: 10.1002/acm2.14527. Epub 2024 Sep 16.

Brain tumor segmentation by combining MultiEncoder UNet with wavelet fusion

Yuheng Pan  1 , Haohan Yong  1 , Weijia Lu  1 , Guoyan Li  1 , Jia Cong  1
Affiliations

Brain tumor segmentation by combining MultiEncoder UNet with wavelet fusion

Yuheng Pan et al. J Appl Clin Med Phys. 2024 Nov.

Abstract

Background and objective: Accurate segmentation of brain tumors from multimodal magnetic resonance imaging (MRI) holds significant importance in clinical diagnosis and surgical intervention, while current deep learning methods cope with situations of multimodal MRI by an early fusion strategy that implicitly assumes that the modal relationships are linear, which tends to ignore the complementary information between modalities, negatively impacting the model's performance. Meanwhile, long-range relationships between voxels cannot be captured due to the localized character of the convolution procedure.

Method: Aiming at this problem, we propose a multimodal segmentation network based on a late fusion strategy that employs multiple encoders and a decoder for the segmentation of brain tumors. Each encoder is specialized for processing distinct modalities. Notably, our framework includes a feature fusion module based on a 3D discrete wavelet transform aimed at extracting complementary features among the encoders. Additionally, a 3D global context-aware module was introduced to capture the long-range dependencies of tumor voxels at a high level of features. The decoder combines fused and global features to enhance the network's segmentation performance.

Result: Our proposed model is experimented on the publicly available BraTS2018 and BraTS2021 datasets. The experimental results show competitiveness with state-of-the-art methods.

Conclusion: The results demonstrate that our approach applies a novel concept for multimodal fusion within deep neural networks and delivers more accurate and promising brain tumor segmentation, with the potential to assist physicians in diagnosis.

Keywords: 3D discrete wavelet transformer; brain tumor segmentation; multi‐encoder.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflicts of interest.

Figures

FIGURE 1
FIGURE 1
A sample from the BraTS2021 training dataset is depicted, exhibiting images arranged in the following order: T1, T1ce, T2, FLAIR, and the Ground Truth, which embodies the doctor's manual segmentation annotation. The color codes red, blue, and green correspond to the representation of necrotic tissue, enhancing tumor and edema, respectively.
FIGURE 2
FIGURE 2
Early fusion and late fusion (early fusion is the combined input of multiple modalities into a network, and late fusion is the separate input of multiple modalities into networks).
FIGURE 3
FIGURE 3
Overview of our proposed architecture. The whole network has five levels. In the first three layers, only residual blocks (grey) extract features, and in the last two layers, a global context‐aware module (orange) captures global information. The wavelet fusion module (green) fuses the modal features extracted by multiple encoders and concat the fused features to the decoder.
FIGURE 4
FIGURE 4
The 3D wavelet fusion module.
FIGURE 5
FIGURE 5
The 3D global context‐aware module.
FIGURE 6
FIGURE 6
Visual segmentation results of ablation experiments. The non‐enhancing area is red, green indicates edema, and blue represents the enhancing tumor.
FIGURE 7
FIGURE 7
The segmentation results for several approaches are visualized. Red: non‐enhancing area, Green: edema, Blue: enhancing tumor.
FIGURE 8
FIGURE 8
Comparison of the convergence speed of different model loss curves at 100 epochs.
FIGURE 9
FIGURE 9
Boxplots are utilized to facilitate the comparison of various methods concerning both Dice metrics and Hausdorff95 Distance metrics. Dots show outliers.
FIGURE 10
FIGURE 10
Boxplot and scatter plots of Dice coefficients. (a) and (b) are from the BraTS2018 online validation set, and (c) and (d) are from the BraTS2021 online validation set.
See this image and copyright information in PMC

References

    1. Pereira S, Pinto A, Alves V, Silva CA. Brain tumor segmentation using convolutional neural networks in MRI images. IEEE Trans Med Imaging. 2016;35(5):1240‐1251. - PubMed
    1. Choi SG, Sohn CB. Detection of HGG and LGG brain tumors using U‐Net. Medico‐legal Update. 2019;19(1):560‐565.
    1. Shukla G, Alexander GS, Bakas S, et al. Advanced magnetic resonance imaging in glioblastoma: a review. Chin Clin Oncol. 2017;6(4):40. - PubMed
    1. Bakas S, Reyes M, Jakab A, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. [published online ahead of print April 23, 2019]. Computer Vision and Pattern Recognition. doi:10.48550/arXiv.1811.02629 - DOI
    1. Yue W, Wang Z, Tian B, Pook M, Liu X. A hybrid model‐and memory‐based collaborative filtering algorithm for baseline data prediction of Friedreich's ataxia patients. IEEE Trans Ind Inf. 2020;17(2):1428‐1437.

MeSH terms