Dissection of tumoral niches using spatial transcriptomics and deep learning

Abstract

This study introduces TG-ME, an innovative computational framework that integrates transformer with graph variational autoencoder (GraphVAE) models for dissection of tumoral niches using spatial transcriptomics data and morphological images. TG-ME effectively identifies and characterizes niches in bench datasets and a high resolution NSCLC dataset. The pipeline consists in different stages that include normalization, spatial information integration, morphological feature extraction, gene expression quantification, single cell expression characterization, and tumor niche characterization. For this, TG-ME leverages advanced deep learning techniques that achieve robust clustering and profiling of niches across cancer stages. TG-ME can potentially provide insights into the spatial organization of tumor microenvironments (TME), highlighting specific niche compositions and their molecular changes along cancer progression. TG-ME is a promising tool for guiding personalized treatment strategies by uncovering microenvironmental signatures associated with disease prognosis and therapeutic outcomes.

Keywords: Artificial intelligence; Cancer; Microenvironment; Transcriptomics.

Graphical abstract

Figure 1

Quantitative evaluation of multiple methods in the DLPFC dataset (A) Pathologist annotation. (B) TG-ME prediction. (C) STAGATE prediction. (D) SEDR prediction. (E) SpaGCN prediction. (F) Leiden algorithm prediction. (G) Comparison of ARI and NMI scores (ranging from 0 to 1) across different methods.

Figure 2

Quantitative evaluation of multiple methods in the breast cancer dataset (A) Pathologist annotation. (B) TG-ME prediction. (C) STAGATE prediction. (D) SEDR prediction. (E) SpaGCN prediction. (F) Leiden algorithm prediction. (G) Comparison of ARI and NMI scores (ranging from 0 to 1) across different methods.

Figure 3

Quantitative evaluation of multiple methods in the breast cancer dataset (A) Pathologist annotation. (B) TG-ME prediction. (C) STAGATE prediction. (D) SEDR prediction. (E) SpaGCN prediction. (F) Leiden algorithm prediction. (G) Comparison of ARI and NMI scores (ranging from 0 to 1) across different methods.

Figure 4

Ablation scores of the multimodal data for TG-ME normalization (A) Pathologist annotation. (B) TG-ME prediction with all the data. (C) Ablation of the gene expression. (D) Ablation of the morphology images. (E) Ablation of the spatial location. (F) Ablation score of the cell composition.

Figure 5

Major steps of the TG-ME pipeline (A) Pre-processing of gene expression. (B) Division into cells. (C) Location based adjacency matrix. (D) Gene expression and cell type composition. (E) Morphological feature extraction. (F) Normalized expression. (G) TG-ME model. (H) Identification of niches from clustering. (I) Interpretation and spatial mapping. (J) TME composition across regions. (K) Severity signatures across regions.

Figure 6

TG-ME model overview (A) The transformer model. (B) The graph variational autoencoder model.

Figure 7

Ablation scores of the TG-ME model (A) Pathologist annotation. (B) TG-ME prediction with both transformer and GraphVAE modules. (C) Ablation of the GraphVAE module. (D) Ablation of the transformer module. (E) Ablation of both GraphVAE and transformer modules, replaced by a fully connected network.

Figure 8

Examination of cell typing of each patient in UMAP and selected gene markers (A) Integrated UMAP visualization of four samples. (B) UMAP visualization of the individual samples. (C) Cell type marker genes.

Figure 9

TG-ME improves the separation of cell types (A) Sample 1 with gene expression only. (B) Sample 1 with TG-ME integration. (C) Sample 4 with gene expression only. (D) Sample 4 with TG-ME integration. (E) Sample 3 before TG-ME integration. (F) Sample 3 after TG-ME integration, where the tumor cells separate into two clusters, and the rest of the cluster becomes well defined. (G) Sample 2 before TG-ME integration. (H) Sample 2 after TG-ME integration, where the tumor cells separate into two clusters, and the rest become well-defined.

Figure 10

Tumor microenvironment identified by TG-ME (A) TME (left) and TME cellular composition (right) for sample 1. (B) TME (left) and TME cellular composition (right) for sample 4. (C) TME (left) and TME cellular composition (right) for sample 3. (D) TME (left) and TME cellular composition (right) for sample 2.