Deep Learning in Neuroradiology

Abstract

Deep learning is a form of machine learning using a convolutional neural network architecture that shows tremendous promise for imaging applications. It is increasingly being adapted from its original demonstration in computer vision applications to medical imaging. Because of the high volume and wealth of multimodal imaging information acquired in typical studies, neuroradiology is poised to be an early adopter of deep learning. Compelling deep learning research applications have been demonstrated, and their use is likely to grow rapidly. This review article describes the reasons, outlines the basic methods used to train and test deep learning models, and presents a brief overview of current and potential clinical applications with an emphasis on how they are likely to change future neuroradiology practice. Facility with these methods among neuroimaging researchers and clinicians will be important to channel and harness the vast potential of this new method.

Fig 1.

Artificial intelligence methods. Within the subset of machine learning methods, deep learning is usually implemented as a form of supervised learning.

Fig 2.

Parallels between artificial and biologic neural networks. Hidden layers of artificial neural networks can be thought to be analogous to brain interneurons.

Fig 3.

Example of a simple deep network architecture. The goal of this network is to classify MR images into 4 specific diagnoses (normal, tumor, stroke, hemorrhage). Multiple different images form the training set. For each new case, the image is broken down into its constituent voxels, each one of which acts as an input into the network. This example has 3 hidden layers with 7 neurons in each layer, and the final output is the probabilities of the 4 classification states. All layers are fully connected. At the bottom is a zoomed-in view of an individual neuron in the second hidden layer, which receives input from the previous layer, performs a standard matrix multiplication (including a bias term), passes this through a nonlinear function (the rectified linear unit function in this example), and outputs a single value to all the neurons of the next layer.

Fig 4.

Example of training and deployment of deep convolutional neural networks. During training, each image is analyzed separately, and at each layer, a small set of weights (convolution kernel) is moved across the image to provide input to the next layer. Each layer can have multiple channels. By pooling adjacent voxels or using larger stride distances between application of the kernel, deeper layers often have smaller spatial dimensions but more channels, each of which can be thought of as representing an abstract feature. In this example, 5 convolutional layers are followed by 3 fully connected layers, which then output a probability of the image belonging to each class. These probabilities are compared with the known class (stroke in the training example) and can be used to measure of how far off the prediction was (cost function), which can then be used to update the weights of the different kernels and fully connected parameters using back-propagation. When the model training is complete and deployed on new images, the process will produce a similar output of probabilities, in which it is hoped that the true diagnosis will have the highest likelihood.

Fig 5.

An example of improving the SNR of arterial spin-labeling MR imaging using deep learning. The model is trained using low-SNR ASL images acquired with only a single repetition, while the reference image is a high-SNR ASL image acquired with multiple repetitions (in this case, 6 repetitions). Proton-density-weighted images (acquired routinely as part of the ASL scans for quantitation) and T2-weighted images are also used as inputs to the model to improve performance. The results of passing the low-SNR ASL image through the model are shown on the right, a synthetic image with improved SNR. In this example, the root-mean-squared error (RSME) between the reference image and the synthetic image compared with the original image is reduced nearly 3-fold, from 29.3% to 10.8%.

Fig 6.

An example of the predicted risk of final infarct for 2 patients with acute ischemic stroke using 2 neural networks trained, respectively, on patients with and without rtPA administration. Patient A is a 76-year-old woman with an admission NIHSS score of 10 scanned 1.5 hours after symptom onset. The +rtPA network estimates a negligible permanent lesion, consistent with the acute DWI suggesting little permanent tissue damage at follow-up. The −rtPA network indicates that without treatment, a considerable volume of the acute ischemic region will progress to a permanent lesion. Patient B is a 72-year-old man also with an admission NIHSS score of 10, scanned 2 hours after onset. In this case, the 2 networks indicate little expected impact of treatment, likely due to the progression at the time of imaging of the ischemic event as seen on the DWI. CMRO₂ indicates cerebral metabolic rate of oxygen; Tmax, time-to-maximum. Figure courtesy of Kim Mouridsen and Anne Nielsen/Aarhus University, Combat Stroke.