Deep-learning in situ classification of HIV-1 virion morphology

Abstract

Transmission electron microscopy (TEM) has a multitude of uses in biomedical imaging due to its ability to discern ultrastructure morphology at the nanometer scale. Through its ability to directly visualize virus particles, TEM has for several decades been an invaluable tool in the virologist's toolbox. As applied to HIV-1 research, TEM is critical to evaluate activities of inhibitors that block the maturation and morphogenesis steps of the virus lifecycle. However, both the preparation and analysis of TEM micrographs requires time consuming manual labor. Through the dedicated use of computer vision frameworks and machine learning techniques, we have developed a convolutional neural network backbone of a two-stage Region Based Convolutional Neural Network (RCNN) capable of identifying, segmenting and classifying HIV-1 virions at different stages of maturation and morphogenesis. Our results outperformed common RCNN backbones, achieving 80.0% mean Average Precision on a diverse set of micrographs comprising different experimental samples and magnifications. We expect that this tool will be of interest to a broad range of researchers.

Keywords: Artificial intelligence; Computer vision; Deep learning; Electron microscopy; HIV-1; Quantitative biology; Virology.

Graphical abstract

Fig. 1

HIV-1 virion morphologies pertinent to this study. A Schematic representation showing the configuration of an immature virion and samples from TEM micrographs. The color scheme cartoons the following components from exteror to interior: blue, envelope glycoproteins; green, lipid bilayer; yellow, matrix protein; black, capsid protein; red, RNA. B Schematic representation showing the configuration of a mature virion and samples from TEM micrographs. C Schematic representation showing the configuration of an eccentric virion and samples from TEM micrographs.

Fig. 2

Network architecture of the deep-learning classifier developed in the present work. The methodology is built upon a two component classifier system that is able to segment the particles (RPN) and classify the segmented regions (CNN). A Schematic of a two-stage Faster RCNN architecture for multiple object detection and classification (instance segmentation). Faster RCNN uses a Region Proposal Network (RPN) to generate Regions of Interest (RoI) for classification. Both RPN and classifier heads share the same backbone CNN. B Our backbone CNN, TEMNet, is composed of several ConvBlocks (Convolution, GroupNormalization and ReLU activation) and MaxPooling layers. We used a Feature Pyramid Network (FPN) to generate multi-scale feature maps on which to generate predictions. Network activations were funneled to three output channels, via two fully-connected layers and soft max output. Each output channel denotes one classification of viral particle.

Fig. 3

Learning error analysis of our particle classifier. A Training and validation error for the ResNet101 backbone versus training epoch, one full cycle of our Faster RCNN implementation. B Training and validation error for the TEMNet backbone versus training epoch. Early stopping and learning rate reduction were used to select the best weights avoiding overfitting. C Ground Truth labeled micrograph. All micrographs used for training our network were evaluated by eye and manually labeled. Instance classification of 200 HIV-1 particles required approximately 30 min to complete. D Automated predictions obtained using the TEMNet backbone for the micrograph in C. The network generated predictions on 130 micrographs in 5 min on one GPU. Scale bars in panels C and D are shown in the lower left portions of the images. Bounding box colors in panels C and D identify viral classifications: mature (yellow), immature (blue), eccentric (green).

Fig. 4

Feature map activations corresponding to each convolutional block of the TEMNet backbone (C1, C2, C3, C4) and each level of the Feature Pyramid Network (P2, P3, P4). The first convolutional block identifies borders and electron density in virion lumina and isolates it from the background, the second block learns image processing techniques such as watershedding. Additional layers become more abstract and escape comprehensive description.

Fig. 5

Micrograph segmentation via a sliding window: A A windowed region was translated across the image and predictions were generated on the segmented regions. B The predictions were gathered on the full scale micrograph and C Non-max suppression (NMS) was applied to determine classifications with highest confidence from overlapping Regions of Interest (RoIs), to glean final predictions. Numbers above each bounding box correspond to prediction “confidence” or certainty, which may ultimately be used to filter predictions (see Fig. 7). Scale bars are shown in the lower left portions of panels B and C.

Fig. 6

Virion classification on multi-magnification micrograph sets. Predictions across different raw TEM micrographs with A The same magnification used for training (30,000$\times$). B A magnification lower than is discernible by a trained expert ($20,000\times$). Our RCNN network calculates the appropriate sliding window size to segment a micrograph according to its magnification. Scale bars are shown beneath the micrographs.

Fig. 7

In situ classifications of virions from different HIV-1 IN (D116N, N184L, delIN) and PR mutant (D25A) viruses. A Ground truth distribution from manually ascribed micrograph sets. B Resulting distributions from TEMNet’s predictions on the same micrographs. Predictions with a confidence score c above 0.5 were counted while those under this confidence threshold were rejected. Numbers over each distribution indicate the number of virus particles counted and the number of independent micrographs analyzed (*). Error bars represent standard deviation from experimental replicates.

Fig. 8

In situ classifications of virions from WT $\mathrm{HIV}-1_{\mathrm{NL}4-3}$, IN deletion mutant delIN, and primary HIV-1 isolates YU2, and JR-CSF. A Ground truth distribution from manually ascribed micrograph sets. B Resulting distributions from TEMNet’s predictions on the same micrographs. Predictions with a confidence score c above 0.5 were counted while those under this confidence threshold were rejected. Other indicators are the same as Fig. 7.