Deep learning for cellular image analysis

Abstract

Recent advances in computer vision and machine learning underpin a collection of algorithms with an impressive ability to decipher the content of images. These deep learning algorithms are being applied to biological images and are transforming the analysis and interpretation of imaging data. These advances are positioned to render difficult analyses routine and to enable researchers to carry out new, previously impossible experiments. Here we review the intersection between deep learning and cellular image analysis and provide an overview of both the mathematical mechanics and the programming frameworks of deep learning that are pertinent to life scientists. We survey the field's progress in four key applications: image classification, image segmentation, object tracking, and augmented microscopy. Last, we relay our labs' experience with three key aspects of implementing deep learning in the laboratory: annotating training data, selecting and training a range of neural network architectures, and deploying solutions. We also highlight existing datasets and implementations for each surveyed application.

Fig. 1 |. Software 2.0 combines data annotations with deep learning to produce intelligent software.

Annotations produced by expert annotators or by a crowd can be used to train deep learning models to extract insights from data. Once trained, these models can be deployed to process new, unannotated data. The human-in-the-loop extension involves the identification of model errors, error correction to produce new training data, and retraining on an updated dataset.

Fig. 2 |. Common mathematical components of deep learning models.

a, Convolutions extract local features in images, and the weights of each filter can be tuned to extract the best feature for a given dataset and task. b, Transfer functions such as those applied by the common rectified linear unit (ReLU) make possible the learning of nonlinear relationships. c, Pooling operations like max pooling downsample to produce spatially coarse feature maps. Deep learning architectures often use iterative rounds of the three operations in a-c to produce low-dimensional representations of images. d, Dilations allow convolutional and pooling kernels to increase their spatial extent while keeping the number of parameters fixed^,. When used correctly, dilations allow classification networks trained on image patches to be used for dense pixel-level prediction. e-i, Modern deep learning models make use of several architectural elements. e, Separable convolutions perform the convolution operation on each channel separately, which reduces the computing power while preserving accuracy^,. f, Residual networks learn the identity mapping plus a small residual and enable the construction of very deep networks. g, Dense networks allow each layer to see every prior layer, which improves error propagation and encourages both feature reuse and parameter efficiency^,. h, Multi-resolution networks allow the classification layers to see both fine and coarse feature maps. i, Through feature pyramids, object-detection models detect objects at distinct length scales^,. j, A plot of the training error during training reveals the relationships among overfitting, underfitting, and model capacity. The tradeoff among these attributes determines which network architectures are suitable for a given task. “Underfitting” refers to models with insufficient representational power, and “overfitting” refers to models that have learned features specific to training data and hence generalize poorly to new, unseen data. Increased model capacity reduces underfitting but can increase the risk of overfitting. k-m, Numerous regularization techniques ensure that deep learning models learn general features from data. k, Batch normalization both regularizes networks and reduces the time needed for training. It was initially created to mitigate covariate shift but was recently found to smooth the landscape of the loss function. l, Dropout randomly turns off filters during training, which regularizes the network by forcing it to not overly rely on any one feature to make predictions. Batch normalization and dropout are typically not used together in the same model. m, L2 regularization penalizes large weights and reduces overfitting.

Fig. 3 |. Image classification applied to biological images.

a, A deep-learning-based image classifier accurately identifies spatial patterns of protein expression in fluorescence images. b, Deep-learning-based image classifiers can accurately interpret changes in cell morphology in imaging-based high-throughput screening. These models are trained on classification tasks and then used to extract feature vectors from images, which can be clustered to identify novel cell phenotypes.

Fig. 4 |. Image segmentation applied to biological images.

a, Instance segmentation identifies every instance of an object type, such as cell nuclei. b, Processing schema for instance segmentation. Pixel-classification approaches attempt to accurately predict object boundaries, deep watershed approaches learn a distance transform, object-detection methods predict a bounding box for each object, and embedding methods assign pixels in different objects to different vectors. c, Application of deep-learning-based image segmentation to spatial proteomics of breast cancer by Keren et al.. Segmentation masks were used to measure signal intensity for each channel in each cell. This information was used by clustering algorithms to identify cell types and cell states. The ability to accurately segment single cells allowed Keren et al. to quantify immune behavior in the tumor microenvironment. Adapted with permission from ref. , Elsevier.

Fig. 5 |. Augmenting microscopy images with deep learning.

a, Deep learning accesses latent data in biological images by using fluorescence images of biological structures as a guide. This strategy yields predictions of fluorescence images and can also be used to improve image quality. b,c, This deep learning model infers which neurons are alive or dead directly from bright-field images. Adapted with permission from ref. , Elsevier.

Computing gradients with backpropagation. a, During the forward pass, local derivatives are computed alongside the original computation. b, During the backward pass, the chain rule is used in conjunction with the local derivatives to compute the derivative of the loss function with respect to each weight.