Image Reconstruction: From Sparsity to Data-adaptive Methods and Machine Learning

Abstract

The field of medical image reconstruction has seen roughly four types of methods. The first type tended to be analytical methods, such as filtered back-projection (FBP) for X-ray computed tomography (CT) and the inverse Fourier transform for magnetic resonance imaging (MRI), based on simple mathematical models for the imaging systems. These methods are typically fast, but have suboptimal properties such as poor resolution-noise trade-off for CT. A second type is iterative reconstruction methods based on more complete models for the imaging system physics and, where appropriate, models for the sensor statistics. These iterative methods improved image quality by reducing noise and artifacts. The FDA-approved methods among these have been based on relatively simple regularization models. A third type of methods has been designed to accommodate modified data acquisition methods, such as reduced sampling in MRI and CT to reduce scan time or radiation dose. These methods typically involve mathematical image models involving assumptions such as sparsity or low-rank. A fourth type of methods replaces mathematically designed models of signals and systems with data-driven or adaptive models inspired by the field of machine learning. This paper focuses on the two most recent trends in medical image reconstruction: methods based on sparsity or low-rank models, and data-driven methods based on machine learning techniques.

Keywords: Compressed sensing; Deep learning; Dictionary learning; Efficient algorithms; Image reconstruction; MRI; Machine learning; Multi-layer models; Nonconvex optimization; PET; SPECT; Sparse and low-rank models; Structured models; Transform learning; X-ray CT.

Fig. 1.

Fundamental duality between sparsity in the image domain and low-rank Hankel matrix in Fourier domain.

Fig. 2.

The synthesis dictionary model for image patches: overlapping patches P_jx of the image x are assumed approximated by sparse linear combinations of columns of the dictionary D, i.e., P_jx ≈ Dz_j, where z_j has several zeros (denoted with white blocks above).

Fig. 3.

Dictionary Learning for MRI (images from [122]): (a) SOUP-DILLO MRI [122] reconstruction (with ℓ₀ penalty) of the water phantom [102]; (b) sampling mask in k-space with 2.5x undersampling; and (c) real and (d) imaginary parts of the dictionary learned during reconstruction, with atoms shown as 6 × 6 patches.

Fig. 4.

Cone-beam CT reconstructions (images from [26]) of the XCAT phantom [135] using the FDK, PWLS-EP [136] (with edge-preserving regularizer), and PWLS-ULTRA [26] (K = 15) methods at dose I₀ = 5 × 10³ incident photons per ray, shown along with the ground truth (top left). The central axial, sagittal, and coronal planes of the 3D reconstruction are shown. The learning-based PWLS-ULTRA removes noise and preserve edges much better than the other schemes.

Fig. 5.

MRI reconstructions (images from [128]) with pseudo-radial sampling and 5x undersampling using Sparse MRI [15] (PSNR = 27.92 dB), ADMM-Net [142] (PSNR = 30.67 dB), and STROLLR-MRI [140] (PSNR = 31.98 dB), along with the original image from [142]. STROLLR-MRI clearly outperforms the nonadaptive Sparse MRI, while ADMM-Net also produces undesirable artifacts.

Fig. 6.

The reconstruction model (see [148]) derived from the image update step of the square transform learning-based image reconstruction algorithm in [29]. The model here has K layers corresponding to K iterations. Each layer first has a decorruption step that computes the second term in (21) using filtering and thresholding operations, assuming a transform model with L filters. This is followed by a system model block that adds the fixed bias term νA^Hy to the output of the decorruption step and performs a least-squares type image update (e.g., using CG) to enforce the imaging forward model.

Fig. 7.

Various realizations of deep learning for image reconstruction.

Fig. 8.

Left to right: full-dose FBP reconstruction, quarter-dose FBP reconstruction, and WavResNet denoising results [159] applied to 25% dose FBP images. The detailed textures, vessel structure, and cancer lesions are clearly seen in the denoised results.

Fig. 9.

An example of encoder-decoder CNN. The encoder is composed of the first κ layers, while the latter κ layers form a decoder network.

Fig. 10.

A high level illustration of input space partitioning for the three layer neural network with two filter channels for each layer. Input images at each partition share the same linear representation, but not across different partitions.