Multi-Modal Brain Tumor Data Completion Based on Reconstruction Consistency Loss

Abstract

Multi-modal brain magnetic resonance imaging (MRI) data has been widely applied in vison-based brain tumor segmentation methods due to its complementary diagnostic information from different modalities. Since the multi-modal image data is likely to be corrupted by noise or artifacts during the practical scanning process, making it difficult to build a universal model for the subsequent segmentation and diagnosis with incomplete input data, image completion has become one of the most attractive fields in the medical image pre-processing. It can not only assist clinicians to observe the patient's lesion area more intuitively and comprehensively, but also realize the desire to save costs for patients and reduce the psychological pressure of patients during tedious pathological examinations. Recently, many deep learning-based methods have been proposed to complement the multi-modal image data and provided good performance. However, current methods cannot fully reflect the continuous semantic information between the adjacent slices and the structural information of the intra-slice features, resulting in limited complementation effects and efficiencies. To solve these problems, in this work, we propose a novel generative adversarial network (GAN) framework, named as random generative adversarial network (RAGAN), to complete the missing T1, T1ce, and FLAIR data from the given T2 modal data in real brain MRI, which consists of the following parts: (1) For the generator, we use T2 modal images and multi-modal classification labels from the same sample for cyclically supervised training of image generation, so as to realize the restoration of arbitrary modal images. (2) For the discriminator, a multi-branch network is proposed where the primary branch is designed to judge whether the certain generated modal image is similar to the target modal image, while the auxiliary branch is to judge whether its essential visual features are similar to those of the target modal image. We conduct qualitative and quantitative experimental validations on the BraTs2018 dataset, generating 10,686 MRI data in each missing modality. Real brain tumor morphology images were compared with synthetic brain tumor morphology images using PSNR and SSIM as evaluation metrics. Experiments demonstrate that the brightness, resolution, location, and morphology of brain tissue under different modalities are well reconstructed. Meanwhile, we also use the segmentation network as a further validation experiment. Blend synthetic and real images into a segmentation network. Our segmentation network adopts the classic segmentation network UNet. The segmentation result is 77.58%. In order to prove the value of our proposed method, we use the better segmentation network RES_UNet with depth supervision as the segmentation model, and the segmentation accuracy rate is 88.76%. Although our method does not significantly outperform other algorithms, the DICE value is 2% higher than the current state-of-the-art data completion algorithm TC-MGAN.

Keywords: Adversarial generation network; Brain tumor segmentation; Image synthesis; Multimodality; Reconstruction consistency.

Fig. 1

Image translation using a Pix2Pix/CycleGAN (1-to-1), b CollaGAN (n-to-1), c TC-MAGN/OURS (RAGAN) (1-to-n). In multi-domain image completion, OUR (RAGAN) completes the missing domain image of the T2 domain image given in the input

Fig. 2

Overview of the proposed 1-to-n multi-domain completion framework. RAGAN (Random Generative Adversarial Networks) is composed of a discriminator D and two generator G. Through training G, it can transform the modal image x into the modal image y, namely G(x, c) = y, can randomly select the target modal label c during the training process so that G can learn how to flexibly transform into different modal images, and the discriminator D can realize the input, and the data distinguishes the true and false function, and the auxiliary classifier is introduced into D

Fig. 3

Overview of the network structure of the generator. In order to ensure that the converted image retains the brain tumor feature information of the input image, we designed a generator with a cyclic consistency structure

Fig. 4

Overview of the discriminator network structure. The discriminator D that introduces the auxiliary classifier can discriminate the authenticity of the input data, so that it can control multiple categories

Fig. 5

The line graph of the change of the pixel-wise similarity between the generated images and the ground truth images with the number of epochs during the training process

Fig. 6

Comparing the image effects synthesized by different methods, compared with other methods, the image generated by our proposed method not only has the similarity at the pixel level in the image, but also maintains better semantic consistency with the ground truth

Fig. 7

Image effects synthesized by different methods. Compared with other data synthesis algorithms, the image generated by our proposed method not only has image pixel-level similarity, but also maintains better semantic consistency with ground truth

Fig. 8

The introduction of cyclic consistency loss preserves the characteristic information of brain tumors to the greatest extent and generates data that is very similar to the original modal image

Fig. 9

Comparing the training effects of images synthesized by different methods, our method can retain tumor feature information to a greater extent, and the segmentation results are closer to the segmentation label

Fig. 10

Comparing the impact of different modal data on the segmentation results, multi-modal data not only helps to improve the accuracy of tumor segmentation, but also improves the network’s distinction between different substantive regions