Incorporating prior knowledge via volumetric deep residual network to optimize the reconstruction of sparsely sampled MRI

Abstract

For sparse sampling that accelerates magnetic resonance (MR) image acquisition, non-linear reconstruction algorithms have been developed, which incorporated patient specific a prior information. More generic a prior information could be acquired via deep learning and utilized for image reconstruction. In this study, we developed a volumetric hierarchical deep residual convolutional neural network, referred to as T-Net, to provide a data-driven end-to-end mapping from sparsely sampled MR images to fully sampled MR images, where cartilage MR images were acquired using an Ultra-short TE sequence and retrospectively undersampled using pseudo-random Cartesian and radial acquisition schemes. The network had a hierarchical architecture that promoted the sparsity of feature maps and increased the receptive field, which were valuable for signal synthesis and artifact suppression. Relatively dense local connections and global shortcuts were established to facilitate residual learning and compensate for details lost in hierarchical processing. Additionally, volumetric processing was adopted to fully exploit spatial continuity in three-dimensional space. Data consistency was further enforced. The network was trained with 336 three-dimensional images (each consisting of 32 slices) and tested by 24 images. The incorporation of a priori information acquired via deep learning facilitated high acceleration factors (as high as 8) while maintaining high image fidelity (quantitatively evaluated using the structural similarity index measurement). The proposed T-Net had an improved performance as compared to several state-of-the-art networks.

Figure 1

The workflow of deep learning based MR reconstruction. MR images were retrospectively undersampled in k-space and transformed back to the image domain. A deep convolutional neural network was trained to provide a mapping from sparsely sampled zero-filled images to fully sampled high quality images, where the loss between predicted images and ground truth images was back-propagated and used to update model parameters. The trained network model was employed to predict high quality images from undersampled test images, whose output subsequently passed through data consistency enforcement to form the final reconstructed images.

Figure 2

Two k-space undersampling patterns retrospectively applied in this study. (a) CIRCUS: a pseudo-random variable-density Cartesian sampling pattern, where sparse sampling was performed on the ky-kz plane. (b) stack-of-stars radial sampling, where undersampling was conducted on the kx-ky plane.

Figure 3

The hierarchical architecture of the proposed deep convolutional neural network, T-Net. It was composed of a contracting path (on the left) and a subsequent expanding path (on the right). Along the contracting path, the resolution of feature maps shrank at the next level, and the number of feature maps or convolutional kernels doubled as indicated. Along the expanding path, the resolution of feature maps expanded at the subsequent level, and the number of feature maps halved as indicated. In this way, image features were extracted and reorganized at multiple levels with different resolutions. Global shortcut connections were established between the corresponding levels of the two paths, whereas local shortcut connections were constructed within the same level of a single path. Finally, after convolving with a 1×1×1 kernel, output MR images were predicted.

Figure 4

Comparison of the proposed local shortcut connection scheme in T-Net with the ones adopted in U-Net and V-Net. (a) no local shortcuts, as in U-Net (b) simple local shortcuts (forwarding the input of a hierarchical level to the output at the same level), as in V-Net (c) relatively dense local shortcuts (forwarding the input of a hierarchical level to all the subsequent convolutional blocks at the same level), as proposed in T-Net.

Figure 5

Comparison of pre-addition activation and post-addition activation in residual learning. The pre-addition activation scheme was adopted in T-Net with identity mapping applied after addition, which was reported to provide faster error reduction and lower training loss than the conventional post-addition activation scheme.

Figure 6

Comparison of zero filled, fully sampled, T-net reconstructed, and compressed sensing images, which were retrospectively undersampled in the CIRCUS pseudo-random Cartesian acquisition scheme with an acceleration factor of 4 achieved. Each row represented an individual subject. The micro-structures lost in the zero-filled images were significantly recovered in the T-net reconstructed images, which had high fidelity with the ground truth. Furthermore, the image quality of the T-Net reconstructed images was improved as compared to that of the compressed sensing images.

Figure 7

Comparison of zero filled, fully sampled, T-net reconstructed, and compressed sensing images, which were retrospectively undersampled in the stack-of-stars radial acquisition scheme with an acceleration factor of 4 achieved. Each row represented an individual subject. The undersampling streak artifacts appearing in the zero filled images were significantly suppressed in the T-Net reconstructed images, which was more consistent with the ground truth. Additionally, the image quality of the T-Net reconstructed images was improved as compared to that of the compressed sensing images.

Figure 8

Comparison of T-net reconstructed, compressed sensing, and zero filled images, which were obtained via retrospective undersampling in the CIRCUS pseudo-random Cartesian acquisition pattern with different acceleration factors achieved. As the acceleration factor was increased from 4 to 6 and 8, the quality of T-Net reconstructed images was slightly degraded with some micro-structures hard to differentiate (in the posterior regions of the knee). However, substantial details were recovered from the very blurry zero filled images. Even when compared with the compressed sensing images, the T-Net reconstructed images still had more high frequency details, improved SNR and suppressed artifacts.

Figure 9

Comparison of T-net reconstructed, compressed sensing, and zero filled images, which were obtained via retrospective undersampling in the stack-of-stars radial acquisition pattern with different acceleration factors achieved. As the acceleration factor was increased from 4 to 6 and 8, global undersampling streak artifacts and local blurring (loss of micro-structures in the posterior regions of the knee) became more obvious across all images. In the T-Net reconstructed images, substantial high frequency details were recovered with SNR increased and streak artifacts suppressed. The T-Net reconstructed images had apparently improved image quality as compared to the compressed sensing images.

Figure 10

The SSIM of images acquired with (a) the CIRCUS Cartesian sampling and (b) the stack-of-stars radial sampling. For a given acceleration factor of 4, the SSIM of T-Net reconstructed images was higher than that of zero filled images and compressed sensing images. The SSIM of the radial acquisition was slightly higher than that of the Cartesian acquisition.

Figure 11

Comparison of images reconstructed using hierarchical deep neural networks with different shortcut patterns, when retrospective Cartesian undersampling was applied with an acceleration factor of 4. (a) ground truth, (b) image reconstructed without local shortcuts (as in U-Net), (c) image reconstructed with simple local shortcuts (as in V-Net), (d) image reconstructed with relatively dense local shortcuts (as in T-Net). The image reconstructed with relatively dense local shortcuts was the most similar to the ground truth.

Figure 12

Comparison of images reconstructed using T-Net with or without data consistency enforcement. (a) ground truth, (b) images reconstructed without data consistency enforcement, and (c) images reconstructed with data consistency enhancement. The data consistency enforcement helped to improve the image quality. The undersampling artifacts were reduced, and the lost micro-structures were better recovered.

Figure 13

Several dense shortcut connection schemes that motivated the design of T-Net. (a) T-Net, in which the input of a network level was forwarded to all the subsequent convolutional blocks at the same level, (b) Dense Net, in which the output of every convolutional block was forwarded to all the subsequent blocks, and (c) Deep Recursive Residual Network, which had shortcut connections with various ranges of influence. Here, the origins of the shortcuts were close to the input of the network level.