A bird's-eye view of deep learning in bioimage analysis

Abstract

Deep learning of artificial neural networks has become the de facto standard approach to solving data analysis problems in virtually all fields of science and engineering. Also in biology and medicine, deep learning technologies are fundamentally transforming how we acquire, process, analyze, and interpret data, with potentially far-reaching consequences for healthcare. In this mini-review, we take a bird's-eye view at the past, present, and future developments of deep learning, starting from science at large, to biomedical imaging, and bioimage analysis in particular.

Keywords: Artificial neural networks; Bioimage analysis; Computer vision; Deep learning; Microscopy imaging.

Fig. 1

Explosive growth of the scientific literature on deep learning and related topics. The graph shows the number of publications per year in the past decade, having the terms deep learning (DL), machine learning (ML), or artificial intelligence (AI) in the title, according to Google Scholar (GS) and Web of Science (WOS) around the time of submission of this article.

Fig. 2

Impact of deep learning on biomedical imaging. The graphs show the number of peer-reviewed journal and selected conference proceedings publications on deep learning in different biomedical application areas, categorized by imaging modality (top, see text for abbreviations) and subject of study (bottom), ranked from most to least popular. Numbers were estimated from the PubMed database of the US National Library of Medicine, National Institutes of Health, around the time of submission of this article, by searching for publications having relevant terms in the title or abstract (Supplementary Data).

Fig. 3

Common tasks in bioimage analysis. The ultimate goal is to gain knowledge of biological processes in health and disease by extracting relevant information from microscopy image or video recordings of these processes. Depending on the specific application, information extraction may involve image enhancement, object detection, image segmentation, object tracking, quantification, and classification, data visualization and analytics, and mathematical or statistical modeling. Deep learning is used increasingly in many of these tasks and we discuss several prominent ones in the main text. The diagram shows a typical order of tasks, with double-headed arrows indicating the possible interrelation and feedback between tasks, as well as the fact that any of them independently may also contribute to knowledge along the way, affecting other tasks. Modified from .

Fig. 4

Examples of successful application of deep learning in bioimage analysis. A: Prediction of a fluorescence microscopy image (middle) from a bright-field microscopy image (left) compared to the truth (right) . The image shows neurons in a culture of induced pluripotent stem cells differentiated toward the motor neuron lineage but containing other cell types as well. Fluorescent labels are TuJ1 (green) with Hoechst (blue) for the cell nuclei. The predicted image was obtained using a multiscale CNN inspired by U-Net. B: Detection of cells in various types of microscopy images : Ki-67 stained bright-field microscopy image of neuroendocrine tumor tissue (top left), phase-contrast microscopy image of HeLa cervical cancer cells (top right), and H&E stained bright-field microscopy images of breast cancer tissue (bottom left) and human bone marrow tissue (bottom right). Detected cells are marked by yellow dots with green circles indicating the ground truth and were obtained using a structured regression model based on a fully residual CNN. C: Segmentation of neuronal axons (blue) and myelin sheaths (red) in a full scanning electron microscopy image slice of a rat spinal cord . The segmentation was obtained using a CNN called AxonDeepSeg. D: Motion analysis of tracked breast cancer susceptibility gene BRCA2 particles in time-lapse fluorescence microscopy images . Tracks were segmented into tracklets showing consistent motion (no switching between different dynamics states) using an LSTM network. Subsequent moment scaling spectrum (MSS) analysis of the tracklets yielded an estimate of the number of mobility classes (three in this case) and their associated parameters. Color coding indicates the value of the MSS slope per tracklet. E: Classification of fluorescence microscopy images (examples at the top) of yeast cells expressing GFP-tagged proteins localizing to 15 subcellular compartments . The classification was done using a CNN called DeepLoc. A visualization (bottom) of the activations of the final convolutional layer of the network in 2D using t-distributed stochastic neighbor embedding (t-SNE) illustrates the power of the model to distinguish the different classes. For more detailed information, see the cited papers, from which the shown examples were adapted with permission (see Acknowledgments section). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)