Prospective Deployment of Deep Learning in MRI: A Framework for Important Considerations, Challenges, and Recommendations for Best Practices

Abstract

Artificial intelligence algorithms based on principles of deep learning (DL) have made a large impact on the acquisition, reconstruction, and interpretation of MRI data. Despite the large number of retrospective studies using DL, there are fewer applications of DL in the clinic on a routine basis. To address this large translational gap, we review the recent publications to determine three major use cases that DL can have in MRI, namely, that of model-free image synthesis, model-based image reconstruction, and image or pixel-level classification. For each of these three areas, we provide a framework for important considerations that consist of appropriate model training paradigms, evaluation of model robustness, downstream clinical utility, opportunities for future advances, as well recommendations for best current practices. We draw inspiration for this framework from advances in computer vision in natural imaging as well as additional healthcare fields. We further emphasize the need for reproducibility of research studies through the sharing of datasets and software. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY STAGE: 2.

Keywords: MRI reconstruction; artificial intelligence; classification; convolutional neural networks; deep learning; segmentation.

Figure 1:

An overview of the three generalized use-cases of deep learning (DL) and convolutional neural network (CNNs) in MRI. The first category relies on model-free contrast synthesis that synthesizes one image contrast or artifact status into another (a). A model-based reconstruction uses k-space data and embeds physics of Fourier encoding and data consistency into an iterative deep learning model to act as an additional regularizer (b). Images can be input into DL networks for assessment of imaging abnormalities (classification) or for segmentation specific tissues (c).

Figure 2:

A residual-based CNN uses a low-quality input image (a low-resolution image in this case) and tries to create a sparse difference image between the low-resolution and high-resolution image. The sparsity of the model output limits the creation of hallucinated artifacts. A computation of the loss is performed by comparing the high-resolution ground truth to the sum of the low-resolution input and the sparse residual images.

Figure 3:

Example images from a super-resolution technique used for threefold improving the resolution of sagittal double-echo steady-state knee MRI scans. Ground-truth original high-resolution (a,d), super-resolution images (b,e), and tricubic interpolated images (c,f) of axial (a-c) and coronal (d-f) reformations show the improvement that super-resolution brings about, especially in the case of subtle features such as cartilage (dashed arrow), thin collateral ligaments (dotted arrow), and effusion (solid arrow).

Figure 4:

Enlarged views of axial slices from ground-truth (a,b), down-sampled (b,c), spline up-sampled (e,f) and deep-learning super-resolution scans (g,h) in the primary visual cortex, with reconstructed gray matter-white matter surfaces and gray matter-cerebrospinal fluid (CSF) surfaces visualized as colored contours. All scans are 0.7mm isotropic resolution nominally, with the exception of the down-sampled scan that has a resolution of 1.0mm. Reconstructed surfaces displayed in (i,j) are overlaid on the ground-truth 0.7-mm isotropic T1-weighted images. Magenta arrowheads highlight locations where super-resolution images provided improved cortical surface estimates.

Figure 5:

To reduce the total number of breath-holds for 2D cardiac cine, a parallel imaging acquisition is typically used to enable 2-fold scan time acceleration (a). The segmentations of the left ventricular endocardial (blue) and epicardial (yellow) borders, and the right ventricular endocardial border (red) are shown for parallel imaging (b). With model-based DL reconstructions, 12-fold scan time acceleration can be used to achieve single breath-hold scan times without significantly impacted image quality (c). Additionally, DL-accelerated cardiac cine scans can help to reduce inevitable variations between breath-holds which can bias ventricular volume measurements in multi-breath-hold scans (d).

Figure 6:

Comparisons of model-based reconstruction networks, with example input zero-filled, real-valued convolutions only, complex-valued convolutions, compressed sensing (CS) with L1-wavelet regularization, and ground truth fully-sampled reconstructions for undersampling factors (R) of 2.3 and 7.40. The difference maps, scaled by a factor of 40, are displayed under each reconstruction. The complex-valued magnitude images appear sharper than the real-valued images and with reduced noise than the CS images. The complex-valued phase maps appear smoother, with red arrows indicating differences in visibility of subtle details.

Figure 7:

Segmentation of femoral cartilage from knee double-echo steady-state scans shows that the accuracy of the model on an identical test set evaluated with a Dice Score Coefficient (DSC) follows a power-law relationship with the extent of training data with very high correlation coefficients. A power law function for y=f(x) is defined as y = α*x^β.

Figure 8:

Case examples from a post-gadolinium-enhanced T1-weighted image series (a) for the MRI sequence DropOut method showing the segmentation predictions using (b) DeepLab V3 and (c) the proposed method. Segmentation predictions are shown as voxel-wise probability maps (ranging from 0.5 to 1) and performance maps classified as true negative, false positive, and false negative as specified by the color code. The first row shows a 53-year-old female patient with malignant melanoma, while the second and third row show a 55-year-old female with malignant melanoma. The arrows indicate true positive lesions (blue) and false positive lesions (yellow). Note that in the last row, the DeepLab V3 show false positive lesions which are not reported by the DropOut method. Figure courtesy Dr. Endre Grøvik from the ongoing TREATMENT clinical study (NCT03458455).