Brain MR Atlas Construction Using Symmetric Deep Neural Inpainting

Abstract

Modeling statistical properties of anatomical structures using magnetic resonance imaging is essential for revealing common information of a target population and unique properties of specific subjects. In brain imaging, a statistical brain atlas is often constructed using a number of healthy subjects. When tumors are present, however, it is difficult to either provide a common space for various subjects or align their imaging data due to the unpredictable distribution of lesions. Here we propose a deep learning-based image inpainting method to replace the tumor regions with normal tissue intensities using only a patient population. Our framework has three major innovations: 1) incompletely distributed datasets with random tumor locations can be used for training; 2) irregularly-shaped tumor regions are properly learned, identified, and corrected; and 3) a symmetry constraint between the two brain hemispheres is applied to regularize inpainted regions. Henceforth, regular atlas construction and image registration methods can be applied using inpainted data to obtain tissue deformation, thereby achieving group-specific statistical atlases and patient-to-atlas registration. Our framework was tested using the public database from the Multimodal Brain Tumor Segmentation challenge. Results showed increased similarity scores as well as reduced reconstruction errors compared with three existing image inpainting methods. Patient-to-atlas registration also yielded better results with improved normalized cross-correlation and mutual information and a reduced amount of deformation over the tumor regions.

Fig. 1.

The proposed deep network. Context-aware patch swap propagates boundary high frequencies to fill the hole. The symmetry loss is used to constrain the complete F2I translated image. Of note, the training images include both normal and pathological images with irregular holes.

Fig. 2.

The patch-swap operation. (a) Find the 1×1 boundary patch (most similar) by minimizing d(p_i,q_j). (b) Replace p_i with q_j directly as in [25]. (c) Refine p₁ with its neighboring patches and the most similar patch with an soft attention scheme. (d) Refine p₂ based on the previously refined p₁. We note that this patch-swap operation is carried out in the FCN-based CTE’s feature space.

Fig. 3.

The symmetry constraint: when using patient MRI slices in training, we have the ground truth of the orange region $X_{\{R-\widehat{R}\}}$.

Fig. 4.

Illustration of inpainting results of a few sampled MRI slices with different modalities from three patients.

Fig. 5.

Illustration of inpainting results in comparison to the ground truth.

Fig. 6.

Illustration of comparison of different inpainting methods including partial Conv, GLS, and PatchMatch.

Fig. 7.

Illustration of comparison of GRI and GRI+FCR. Note that the FCR step yielded vivid and visually reasonable results compared with GRI only as visually assessed.

Fig. 8.

Illustration of the ablation study on the use of the symmetry constraint (sym), neighboring (NB) refinement, and incomplete (IC) data training.

Fig. 9.

Registration results from six sample subjects in the atlas space using three modalities. Registered images with and without inpainting are shown in columns four and five. Tumor regions are marked in red boxes.

Fig. 10.

Similarity evaluation between the atlas and volumes from fifteen subject registered to the atlas. Both NCC and MI are shown on all three modalities using either direct registration or registration after inpainting. The middle bar marks the median and the cross marks the mean in the boxplots.