ImmunoAIzer: A Deep Learning-Based Computational Framework to Characterize Cell Distribution and Gene Mutation in Tumor Microenvironment

Abstract

Spatial distribution of tumor infiltrating lymphocytes (TILs) and cancer cells in the tumor microenvironment (TME) along with tumor gene mutation status are of vital importance to the guidance of cancer immunotherapy and prognoses. In this work, we developed a deep learning-based computational framework, termed ImmunoAIzer, which involves: (1) the implementation of a semi-supervised strategy to train a cellular biomarker distribution prediction network (CBDPN) to make predictions of spatial distributions of CD3, CD20, PanCK, and DAPI biomarkers in the tumor microenvironment with an accuracy of 90.4%; (2) using CBDPN to select tumor areas on hematoxylin and eosin (H&E) staining tissue slides and training a multilabel tumor gene mutation detection network (TGMDN), which can detect APC, KRAS, and TP53 mutations with area-under-the-curve (AUC) values of 0.76, 0.77, and 0.79. These findings suggest that ImmunoAIzer could provide comprehensive information of cell distribution and tumor gene mutation status of colon cancer patients efficiently and less costly; hence, it could serve as an effective auxiliary tool for the guidance of immunotherapy and prognoses. The method is also generalizable and has the potential to be extended for application to other types of cancers other than colon cancer.

Keywords: biomarker; cell distribution; deep learning; hematoxylin and eosin (H&E); semi-supervised learning; tumor gene mutation; tumor microenvironment (TME).

Figure 1

Study protocol workflow. (A) ImmunoAIzer includes: a cellular biomarker distribution prediction network (CBDPN) and a tumor gene mutation detection network (TGMDN). (B) CBDPN takes H&E image patches as input and makes predictions of the spatial distribution of CD3 and CD20, pan-cytokeratin (PanCK) and DAPI in TME. (C) TGMDN takes H&E image patches as input and detects adenomatous polyposis coli gene (APC), tumor protein P53 gene (TP53), and kirsten rat sarcoma viral oncogene (KRAS) mutations. (TCGA: The Cancer Genome Atlas; H&E: hematoxylin and eosin)

Figure 2

Semi-supervised structure for the ImmunoAIzer cellular biomarker prediction networks. Note that the generator and discriminator structure in this figure is only schematic, a detailed introduction about the generator structures will be given in the following sections.

Figure 3

CBPDN structure to predict biomarker distributions. Inception blocks A to D are a series of convolutional modules at various resolutions which are used extract features at different levels, and Inception block E is used to promote high dimensional representations to boost prediction performance. Details of the module structure are described in Figure S3.

Figure 4

TGMDN structure to detect tumor gene mutations.

Figure 5

Cellular biomarker prediction results. (A) H&E-stained colon cancer slides: the middle image shows an H&E-stained slide (~5500 × 4000 pixels). (The left and right images show enlarged views.) (B) mIHC image of the same tissue sample. (C) Predictions of our proposed biomarker prediction network. (D) 512 × 512 patch-wise results for our proposed method and UNet, DeepLabV3 and DeepLabV3+ under the semi-supervised structure.

Figure 6

Visualization of the prediction result of a whole slide image (WSI) from the TCGA dataset. (A) H&E stained WSI (filename: TCGA-AZ-4616-01Z-00-DX1). The red arrows indicate pathologists confirmed tumor infiltrating lymphocytes (TILs) clusters. (B) Prediction result acquired using semi-supervised CBDPN.

Figure 7

TCGA dataset validation results. (A) Scatter plot of the PanCK-positive cell proportion: results based on CBDPN versus results based on molecular information provided by TCGA. (B) Scatter plot of the TILs proportion: results based on CBDPN (Note that TILs were labeled by CD3 and CD20 in this study) versus results based on molecular information provided by TCGA. (C) Correlation analysis between results based on CBDPN and results based on TCGA information. (D) Distribution of the number of tiles per case. (E) Races and genders distribution of the cases used in this experiment.

Figure 8

PD-1 distribution analysis with the ImmunoAIzer CBDPN. (A) H&E-stained colon cancer sample slide. (B) Anti-PD-1 immunofluorescence-stained image of the same tissue slide. (C) Cell-specific biomarker distribution image generated by CBDPN. (D) Merged image showing the stained PD-1-positive cells overlaid on our predicted cell-specific biomarker distribution image, the orange area represents PD-1-positive TILs.

Figure 9

Comparative analysis of cell quantification based on semi-supervised CBDPN and commercial Inform Software (Version 2.4). (A) Bar chart comparison of CD3- and CD20-positive cell percentage calculated based on CBDPN and Inform software. (B) Correlation analysis of CD3- and CD20-positive cell percentage calculated based on CBDPN and Inform software. (C) Bar chart comparison of PD-1-expressing TILs percentage calculated based on CBDPN and Inform software. (D) Correlation analysis of PD-1-expressing TILs percentage calculated based on CBDPN and Inform software.

Figure 10

Receiver operating characteristic curves for tumor gene mutation detection network. (A) Receiver operating characteristic curve (ROC) and area under curve (AUC) value for adenomatous polyposis coli gene (APC), tumor protein P53 gene (TP53), and kirsten rat sarcoma viral oncogene (KRAS) mutations. (B) ROC curve and AUC value for APC mutation. (C) ROC curve and AUC value for KRAS mutation. (D) ROC curve and AUC value for TP53 mutation.

Figure 11

Two-dimensional visualization of TGMDN output obtained by implementing the t-SNE Algorithm. (A) Cluster result of APC mutation probability generated by the TGMDN. (B) Cluster result of TP53 mutation probability generated by the TGMDN. (C) Cluster result of KRAS mutation probability generated by the TGMDN. (D) Patch-embedded t-SNE representation with magnifications showing specific mutations that were detected based on the H&E image data from test set that were obtained from the TCGA Colon adenocarcinoma (COAD) project.