Joint Frequency and Image Space Learning for MRI Reconstruction and Analysis

Abstract

We propose neural network layers that explicitly combine frequency and image feature representations and show that they can be used as a versatile building block for reconstruction from frequency space data. Our work is motivated by the challenges arising in MRI acquisition where the signal is a corrupted Fourier transform of the desired image. The proposed joint learning schemes enable both correction of artifacts native to the frequency space and manipulation of image space representations to reconstruct coherent image structures at every layer of the network. This is in contrast to most current deep learning approaches for image reconstruction that treat frequency and image space features separately and often operate exclusively in one of the two spaces. We demonstrate the advantages of joint convolutional learning for a variety of tasks, including motion correction, denoising, reconstruction from undersampled acquisitions, and combined undersampling and motion correction on simulated and real world multicoil MRI data. The joint models produce consistently high quality output images across all tasks and datasets. When integrated into a state of the art unrolled optimization network with physics-inspired data consistency constraints for undersampled reconstruction, the proposed architectures significantly improve the optimization landscape, which yields an order of magnitude reduction of training time. This result suggests that joint representations are particularly well suited for MRI signals in deep learning networks. Our code and pretrained models are publicly available at https://github.com/nalinimsingh/interlacer.

Figure 1:

Maps of correlation coefficients between a single pixel (center of circle) and all other pixels in image (left two panels) and frequency space (right two panels) representations of MNIST and a brain MRI dataset. All maps show strong local correlations useful for inferring missing or corrupted data in both spaces. Frequency space correlations also display conjugate symmetry characteristic of Fourier transforms of real images.

Figure 2:

The Interleaved (left) and Alternating (right) layers, embedded within full network architectures. Each ‘F-Conv’ or ‘I-Conv’ block applies Batch Normalization (BN), a convolution, and an activation function in the frequency or image space, respectively.

Figure 3:

Data generation procedure for undersampling in the presence of motion. At line L_i in frequency space, the original image I is rotated and translated to form $I_i^M$. Lines from the corresponding Fourier transforms F and $F_i^M$ are mixed and undersampled to generate motion-corrupted frequency space data $\tilde{F}$ that would have been acquired under the illustrated motion pattern. A similar method is used to simulate pure motion corruption without undersampling, where all frequency space lines are maintained to generate $\tilde{F}$.

Figure 4:

Subjectwise SSIM comparison for all brain MRI tasks without data consistency constraints. Subjects are sorted by performance of the Interleaved network. For all tasks, networks combining frequency and image space convolutions outperform single-domain networks.

Figure 5:

Example reconstructions with motion at 3% of scanning lines, zoomed-in image patches, difference patches between reconstructions and ground truth images, and frequency space reconstructions. The log values are taken of the frequency space data to better visualize its dynamic range. In the patch difference, red pixels have a higher value in the reconstruction than in the ground truth, while blue pixels have a lower value in the reconstruction than in the ground truth. The Interleaved and Alternating architectures more accurately eliminate the ‘shadow’ of the moved brain and the induced blurring compared to the single-domain networks.

Figure 6:

Example reconstructions from 4x undersampled, motion-corrupted data data, zoomed-in image patches, difference patches between reconstructions and ground truth images, and frequency space reconstructions. As in the motion corruption and undersampling examples, the Interleaved and Alternating architectures provide more accurate reconstructions of the ground truth images and reconstructing a more coherent k-space.

Figure 7:

Example reconstructions with noise of standard deviation 10,000. The Interleaved and Alternating reconstructions remove the pixelated noise effect without over-smoothing, in contrast to the single-domain networks.

Figure 8:

SSIM comparison of the joint networks with the state of the art undersampled reconstruction approaches on ADNI data. Results are reported for three undersampling patterns: 4x uniform undersampling with a fully-sampled central region (left), 8x uniform undersampling with a fully-sampled central region (middle), and 4x undersampling at random (right). In all cases, simple networks composed of repeated copies of our joint layers perform at least as well as other state of the art networks, and in the difficult case of a random sampling pattern, outperform the baseline networks.

Figure 9:

Example reconstructions from 4x undersampled data, with lines selected at random. The Interleaved and Alternating architectures provide more accurate reconstructions of the ground truth images, better eliminating ‘ringing’ and blurring artifacts.

Figure 10:

Example training loss and validation SSIM curves (left) and sample reconstructions and patches for MoDL networks with K = 1, 5 iterations trained with image convolutional layers and with the proposed joint (Interleaved) layers. MoDL networks with image convolutional layers do not converge if trained directly with K = 5. Instead, a K = 1 MoDL network must be trained and used to initialize the weights of a K = 5 MoDL network. MoDL networks trained with joint layers do not require pre-training and achieve the same loss and validation SSIM values as networks trained with image convolutions in significantly less time.

Figure 11:

Typical image reconstruction results for all architectures (rows) and loss functions (columns) on FastMRI images without fat suppression. The Interleaved and Alternating networks provide the sharpest reconstructions for all loss functions. Amongst these, both SSIM-based loss functions most sharply reconstruct high frequency structures within the zoomed-in patch. Similar results are observed in images with fat suppression.

Figure 12:

Comparison of the Interleaved reconstruction results with the top methods on the FastMRI single coil knee reconstruction challenge. All images were taken from the FastMRI online submission website. Our method produces a reconstruction qualitatively similar to those of the top three methods on the leaderboard.