Machine learning for cross-scale microscopy of viruses

Abstract

Despite advances in virological sciences and antiviral research, viruses continue to emerge, circulate, and threaten public health. We still lack a comprehensive understanding of how cells and individuals remain susceptible to infectious agents. This deficiency is in part due to the complexity of viruses, including the cell states controlling virus-host interactions. Microscopy samples distinct cellular infection stages in a multi-parametric, time-resolved manner at molecular resolution and is increasingly enhanced by machine learning and deep learning. Here we discuss how state-of-the-art artificial intelligence (AI) augments light and electron microscopy and advances virological research of cells. We describe current procedures for image denoising, object segmentation, tracking, classification, and super-resolution and showcase examples of how AI has improved the acquisition and analyses of microscopy data. The power of AI-enhanced microscopy will continue to help unravel virus infection mechanisms, develop antiviral agents, and improve viral vectors.

Keywords: CP: Microbiology and CP: Imaging; SARS-CoV-2; adenovirus tracking and trafficking; artificial intelligence; deep learning; electron microscopy; fluorescence super-resolution microscopy; herpes simplex virus; human immunodeficiency virus; influenza virus; machine learning; nanoparticle.

Graphical abstract

Figure 1

Applications of machine learning (ML) in viral image analysis (A) Nuclear segmentation of the Hoechst 33342 DNA signal from human lung epithelial A549 cells infected with AdV using StarDist. The segmented image shows the overlay of the segmentation mask with the raw image. Colors indicate object instance. The white arrow points to an infected nucleus. Scale bar 20 μm. Of note, conventional methods, e.g., Otsu-thresholding or entropy-based methods, are typically unsuccessful in yielding reasonable segmentation masks for all nuclei. (B) Tracking of fluorescently labeled AdV particles containing a capsid tag using TrackMate. Circles in the output show objects detected by differences of Gaussians (DoG), and lines show trajectories obtained from the linear assignment problem (LAP) tracker. Scale bars at the top and bottom indicate 5 and 1 μm, respectively. Movie courtesy of Dr. Michael Bauer. (C) Image denoising using Noise2Void2. The procedure identifies incoming viral DNA genomes in A549 cells infected with ethynyl-deoxy-cytidine (EdC)-tagged AdV-C5 and stained by click chemistry as described. Images show maximum intensity projections of confocal stacks of an EdC staining (image courtesy of Alfonso Gómez-González). The white arrow indicates a virus particle, and the yellow box is a zoomed-in view. Scale bar, 5 μm. (D) Functional classification of AdV-inoculated A549 cells into infected and uninfected cells based on ViResNet and the nuclear signal of the Hoechst dye. Scale bar, 10 μm. (E) An example for data exploration using k-nearest-neighbor (k-NN) clustering.

Figure 2

Applications of deep learning in microscopy DL procedures affect all stages of image acquisition and analysis. Procedures and frameworks include event-driven acquisition, neural network augmented imaging, Fiji, DeepImageJ, CellProfiler, Napari, TrackMate, NucleAIzer, CellPose, StarDist, ilastik, TWS, Pytorch, TensorFlow, Keras, scikit learn, ZeroCostDL4Mic, DeepCell, Bioimage Model Zoo, and Project Jupyter.

Figure 3

How ML enhances viral image analysis (A) Nuclear segmentation of the Hoechst 33342 signal from human lung epithelial A549 cells infected with AdV-C2 expressing GFP from a cytomegalovirus promoter. The input image shows the Hoechst signal. Scale bar, 20 μm. Nuclear segmentation was performed using StarDist or minimum cross-entropy (MCE). For the segmented images, regions in green indicate correctly classified pixels and regions in magenta incorrectly classified pixels with respect to manually curated segmentation maps. White arrow points to an under-segmented nucleus in the StarDist protocol and an over-segmented nucleus in the MCE protocol. The intersection over union (IoU) of two images A and B is calculated as $\mathrm{I}\mathrm{o}\mathrm{U}(\mathrm{A},\mathrm{B})=\frac{\left|\mathrm{A}\cap \mathrm{B}\right|}{\left|\mathrm{A}\cup \mathrm{B}\right|}$, where $\left|\mathrm{A}\cap \mathrm{B}\right|$ is the area of overlap/intersection between A and B, and $\left|\mathrm{A}\cup \mathrm{B}\right|$ is the joint area/union of A and B. IoU values range between 0 and 1, with a higher value indicating a better congruence. (B) Image classification using a convolutional neural network (CNN) as described in Andriasyan et al. Left image shows Hoechst 33342 staining of live A549 cells infected with AdV-C5. Scale bar, 20 μm. Center image shows classification of CNN, with nuclear masks for uninfected cells (yellow) and infected cells (red). Orange arrow points to an incorrectly classified nucleus and white arrow to a nucleus that was not segmented under the chosen probability threshold. The right image shows DAPI nuclear staining (blue) and immunofluorescence (IF) for the viral protein VI expressed late in infection. Of note, although the CNN was trained using a reporter virus that expresses GFP under control of the early-to-late viral IX promoter, the network recognizes features of cells positive for the viral late protein VI indicating network robustness. (C) Image denoising using Noise2Void2. A549-ΔMIB-1 cells were infected with genome-tagged AdV-C5-EdC as described.^, Images show maximum intensity projections of confocal stacks of EdC staining (image courtesy of Alfonso Gómez-González). The scale bars indicate 5 μm for the full-size image and 1 μm for the zoomed-in image. Image smoothing by Gaussian filtering can reduce pixel noise but is less efficient in removing background noise or enhancing the signal from the object of interest. For more comprehensive discussions of image analysis algorithms, metrics, and benchmarking, see Maier-Hein et al., Ulman et al., and Caicedo et al.^,,