Upstream Machine Learning in Radiology

Abstract

Machine learning (ML) and Artificial intelligence (AI) has the potential to dramatically improve radiology practice at multiple stages of the imaging pipeline. Most of the attention has been garnered by applications focused on improving the end of the pipeline: image interpretation. However, this article reviews how AI/ML can be applied to improve upstream components of the imaging pipeline, including exam modality selection, hardware design, exam protocol selection, data acquisition, image reconstruction, and image processing. A breadth of applications and their potential for impact is shown across multiple imaging modalities, including ultrasound, computed tomography, and MRI.

Keywords: Artificial intelligence; Deep learning; Image reconstruction; Medical imaging.

Figure 1:

A summary of the radiology workflow starting from the ordering of an examination and ending with a radiologist interpretation of the acquired imaging. Errors during each of the seven stages of this workflow can lead to delays or the rendering of an incorrect final diagnosis decision.

Figure 2:

Images of simulated channel data comparing cluttered, CNN-filtered, and reference uncluttered images. Reverberation clutter, which appears as a high frequency noise across the channels, is removed by the CNN while preserving the structure of reflections from true targets. Data from Brickson LL, Hyun D, Jakovljevic M, Dahl JJ. Reverberation Noise Suppression in Ultrasound Channel Signals Using a 3D Fully Convolutional Neural Network. IEEE Trans Med Imaging. 2021;PP.

Figure 3:

B-mode images of a longitudinal cross-section of a carotid artery and thyroid. The CNN-filtered image visualizes several hypoechoic and anechoic targets that were originally obscured by clutter. Data from Brickson LL, Hyun D, Jakovljevic M, Dahl JJ. Reverberation Noise Suppression in Ultrasound Channel Signals Using a 3D Fully Convolutional Neural Network. IEEE Trans Med Imaging. 2021;PP.

Figure 4:

CNN-based ultrasound speckle reduction and detail preservation. (left) A traditional B-mode image of a complex focal liver lesion. (right) The CNN output when provided with the same raw data. Data from Hyun D, Brickson LL, Looby KT, Dahl JJ. Beamforming and Speckle Reduction Using Neural Networks. IEEE Trans Ultrason Ferroelectr Freq Control. 2019;66(5):898-910.

Figure 5:

Ultrasound molecular imaging of a breast cancer tumor in a transgenic mouse. (A) B-mode image, with the tumor in the center. (B) Contrast-enhanced image overlaid on the B-mode image. (C) Destructive state-of-the-art molecular image. (D) Output of the trained CNN using only nondestructive input data. The nondestructive CNN closely matched the destructive image, showing the potential for AI-enabled real-time molecular imaging.

Figure 6:

Various technologies for controlling CT dose and image quality are listed under the categories of system design, acquisition, and reconstruction. Sub-figures show (left to right): overall scanner geometry, x-ray spectra from different tube voltages, and two reconstructions with different kernels showing sharper image but higher noise (top) and lower noise but smoother image (bottom).

Figure 7:

CT slice of an abdomen scan showing higher noise at lower dose levels. Mayo patient CT projection data library provides routine (100%) dose and low-dose (25%) after inserting noise in the projection data. The additional four dose levels are synthesized from these two, assuming only quantum noise.

Figure 8:

Non-active and active data strategies for deep learning based data sampling methods. Non-active (fixed) strategies optimize sampling trajectories and reconstruction networks at training time (A). At inference time (B), the sampling trajectory is fixed, and the corresponding gradient waveforms are programmed in the scanner for acquisition. The optimized reconstruction network is then used for reconstructing images from undersampled measurements. Active strategies (C&D) use an additional neural network that suggests the next sample to collect using the reconstruction obtained from existing samples. The process is repeated until a desired metric or uncertainty threshold is met. Due to their sequential nature, active strategies require an additional mechanism that generates gradient waveforms on-the-fly for acquiring the samples proposed by the sample selection network.

Figure 9:

(a) MRI reconstruction model training pipeline. If fully-sampled k-space data is available, a linear reconstruction is performed to generate the high-quality, ground truth image. The fully-sampled k-space data is then retrospectively undersampled by throwing away k-space lines, simulating how the scanner would undersample data in a true, accelerated acquisition. This undersampled data is given to the network for reconstruction. The network is then trained to output a predicted image, which is enforced to be “close” to the ground truth image via a loss function. A simple mean-squared-error loss is shown here. The model parameters are then iteratively updated by a stochastic gradient descent (SGD) algorithm, which intends to minimize the loss function, thereby minimizing the difference between the predicted image and the ground truth image. (b) Once the model is fully trained, it can be used to “infer” or reconstruct images from prospectively undersampled data in an efficient manner.

Figure 10:

(a) An example of a deep convolutional neural network (CNN) for MRI reconstruction. There are many degrees of freedom in designing CNNs. One popular architecture is called a U-Net, which repeatedly applies convolutions and down-sampling layers to extract both low-resolution and high-resolution features. (b) Unrolled neural networks instead apply shallow CNNs (again U-Nets are shown) cascaded with simulated MRI model projections which ensure that the output of each CNN does not deviate from the raw, undersampled k-space data. These projections also make use of Fourier transforms and coil sensitivity information to convert intermediate network outputs back to the original multi-channel k-space domain where the projection is performed.

Figure 11:

Two 2D cardiac cine scans are performed on a pediatric patient with 17 short-axis view (SAX) slices covering the heart and scan parameters: TE=1.4ms, TR=3.3ms, matrix size=200x180. Reformats are shown to visualize 4-chamber (4Ch), 2-chamber (2Ch) views along with a 3D rendering. (a) The first scan is performed with 2X undersampling and reconstructed using a standard parallel imaging technique. (b) The second scan is performed with 12X undersampling and reconstructed using a deep learning approach. With deep learning-powered acceleration, the scan time is shortened from 6 breath-holds down to a single breath-hold. This not only has important implications for patient comfort, but also for the accuracy of volumetric assessments from these images, since the inevitable variations between breath-holds is significantly reduced in the DL images (yellow arrows).

Figure 12:

A conventional supervised learning system (a) and an unsupervised system (b). (a) Framework overview in a supervised setting with a conditional GAN when fully-sampled datasets are available. (b) Framework overview in an unsupervised setting. The input to the generator network is an undersampled complex-valued k-space data and the output is a reconstructed two-dimensional complex-valued image. Next, a sensing matrix comprised of coil sensitivity maps, an FFT and a randomized undersampling mask (drawn independently from the input k-space measurements) is applied to the generated image to simulate the imaging process. The discriminator takes simulated and observed measurements as inputs and tries to differentiate between them.

Figure 13.

Knee application representative results, showing, from left to right: the input undersampled complex image to the generator, the output of the unsupervised generator, the output of the supervised generator, and the fully-sampled image. The acceleration factors of the input image are 6.5, 9.9, and 15.6, from top to bottom. The quantitative metrics that are plotted next to the images are for the slice that is shown. In all rows, the unsupervised GAN has superior PSNR, NRMSE, and SSIM compared to CS. In the first row, the unsupervised GAN has metrics that are notably worse than the supervised GAN. In the middle row and last rows, the unsupervised GAN has metrics that come close to the performance of the supervised GAN.

Figure 14.

The results of the reconstruction performance on the set of knee scans of the unsupervised GAN as a function of the acceleration factor of the training datasets. The y-axis represents PSNR, NRMSE, or SSIM, depending on the plot. The x-axis represents the acceleration factor of the datasets. The gap between PSNR of CS and the unsupervised model is fairly negligible over the range of accelerations. The gap between NRMSE of CS and the unsupervised model is fairly negligible at first, for low accelerations, but becomes more significant at an acceleration of 6 and beyond. The gap between SSIM of CS and the unsupervised model is fairly negligible at first, for an acceleration of 2, but becomes more significant at an acceleration of 4 and beyond.

Figure 15.

2D DCE application representative results, where the left slice is the magnitude of one input undersampled complex image to the generator, the middle slice is the output of the generator and the right slice is a compressed sensing L1-wavelet regularization. The generator greatly improves the input image quality by recovering sharpness and adding more structure to the input images. Additionally, the proposed method produces a sharper reconstruction compared to CS. In the first row, the kidneys of the unsupervised GAN are visibly much sharper than that of the input and CS.