Artificial intelligence with deep learning in nuclear medicine and radiology

Abstract

The use of deep learning in medical imaging has increased rapidly over the past few years, finding applications throughout the entire radiology pipeline, from improved scanner performance to automatic disease detection and diagnosis. These advancements have resulted in a wide variety of deep learning approaches being developed, solving unique challenges for various imaging modalities. This paper provides a review on these developments from a technical point of view, categorizing the different methodologies and summarizing their implementation. We provide an introduction to the design of neural networks and their training procedure, after which we take an extended look at their uses in medical imaging. We cover the different sections of the radiology pipeline, highlighting some influential works and discussing the merits and limitations of deep learning approaches compared to other traditional methods. As such, this review is intended to provide a broad yet concise overview for the interested reader, facilitating adoption and interdisciplinary research of deep learning in the field of medical imaging.

Keywords: Artificial intelligence; Deep learning; Medical imaging; Nuclear medicine; Radiology.

Fig. 1

Growth of AI in radiology reflected by the number of publications on PubMed when searching on the terms “radiology” with “artificial intelligence,” “machine learning” or “deep learning”

Fig. 2

Schematic overview of different machine learning components and their interaction for a brain tumor detection example. A model, defined up to some parameters, receives a brain MRI as input and needs to provide as output whether the brain scan shows a tumor or not. Based on example data, i.e., labeled brain MRI, a learning algorithm optimizes the model parameters to improve a certain performance measure. When training is finished and the model achieves sufficient performance, it can be used to detect tumors in new MRI scans

Fig. 3

Schematic of a fully connected neural network. $\mathit{left}:$ An artificial neuron or perceptron, where the output y is calculated as a sum of weighted inputs $x=[x_1,x_2,...,x_n]$ (with weights $w=[w_1,w_2,...,w_n]$) and a bias b, optionally passed through an activation function f. $\mathit{right}:$ The fully connected neural network is created by connecting these neurons into many layers, where the outputs of one layer serve as the inputs to the following layer. The network depicted here consists of N inputs and M outputs

Fig. 4

Network architecture used in [14] for brain tumor classification in MRI

Fig. 5

Illustration of a convolution operation between a 2D input and a kernel with size = 3 and stride = 1

Fig. 6

Illustration of a residual block

Fig. 7

Transposed convolution operation with a $2\times 2$ kernel and stride 2

Fig. 8

U-Net architecture

Fig. 9

Generative adversarial network (GAN) framework illustrated with a pseudo-CT from MRI generation example

Fig. 10

Architecture of AUTOMAP. Note that the original $n\times n$ k-space data are complex-valued, so that it is reshaped to a vector of size $2n^2$. The convolutional layers use $m_1$ and $m_2$ feature maps, respectively

Fig. 11

Data flow graph for ADMM-CSNet, an unrolled version of the ADMM algorithm used in compressed sensing MRI. The iterative updates $F(\theta )$ are unrolled in a neural network with fixed number of iterations. Each update block $F^n$ can have its own unique parameters ${\theta }^n$, which are learned in an end-to-end fashion

Fig. 12

Illustration of the deep image prior training procedure for dynamic PET denoising. A static image is used as the input $z$ to a network f, initialized with random weights $\theta$. The network parameters are then iteratively optimized to produce the dynamic image x. After a certain number of iterations, denoised versions of the dynamic PET images are obtained as output. Image from [83]

Fig. 13

Schematic overview of a CycleGAN used for synthetic CT generation from MR

Fig. 14

Illustration of the radiomics workflow

Fig. 15

Segmentation examples from the Medical Segmentation Decathlon [134]. a Hepatic vessel (blue) and tumor (green) in CT. b Lung tumor (green) in CT. c Pancreas (blue) and tumor (green) in CT. d Left ventrical (green) in MRI. e Spleen (green) in CT. f Prostate peripheral (blue) and transitional (green) zones in MRI

Fig. 16

Illustration of a typical lung cancer screening pipeline consisting of a lung nodule detection and a malignancy classification stage

Fig. 17

Breast cancer mammography screening using a convolutional neural network. Image adapted from [161] with permission from Elsevier

Fig. 18

Cardiac pathology classification on cine MRI with motion characterization. Image from [173] with permission from Elsevier

Fig. 19

Aneurysm detection network proposed in [200]. Reproduced with permission from The Radiological Society of North America. Image from Yang J, Xie M, Hu C, et al. Deep Learning for Detecting Cerebral Aneurysms with CT Angiography. Radiology 2021;298:155–163