Denoising image-based spatial transcriptomics data with DenoIST

Abstract

Image-based spatial transcriptomics (IST) technologies provide unprecedented resolution of gene expression in tissue sections, but suffer from contamination of cells' gene expression profiles due to imperfect cell segmentation. We present DenoIST (Denoising Image-based Spatial Transcriptomics data), a new computational tool that accurately identifies and removes contaminating transcripts from IST datasets. DenoIST models the observed transcript counts using a Poisson mixture model that explicitly accounts for local neighbourhood contamination. Applied to multiple real IST datasets of varying cell densities, DenoIST restores gene expression specificity and clarifies local biological structures by identifying and filtering transcripts spilt over from neighbouring cells. The denoised data enable more consistent and interpretable cell type annotation by minimising conflicting gene expression profiles, reducing the prevalence of hybrid or ambiguous cell types, and enhancing the contrast between distinct functional compartments. Overall, we demonstrate that DenoIST can be integrated to existing IST analysis workflows to improve biological interpretability and robustness of IST data.