EMBEDR: Distinguishing signal from noise in single-cell omics data

Abstract

Single-cell "omics"-based measurements are often high dimensional so that dimensionality reduction (DR) algorithms are necessary for data visualization and analysis. The lack of methods for separating signal from noise in DR outputs has limited their utility in generating data-driven discoveries in single-cell data. In this work we present EMBEDR, which assesses the output of any DR algorithm to distinguish evidence of structure from algorithm-induced noise in DR outputs. We apply EMBEDR to DR-generated representations of single-cell omics data of several modalities to show where they visually show real-not spurious-structure. EMBEDR generates a "p" value for each sample, allowing for direct comparisons of DR algorithms and facilitating optimization of algorithm hyperparameters. We show that the scale of a sample's neighborhood can thus be determined and used to generate a novel "cell-wise optimal" embedding. EMBEDR is available as a Python package for immediate use.

Keywords: ATAC-seq; UMAP; cell-type identification; clustering; data visualization; dimensionality reduction; quality assessment; single-cell RNA sequencing; single-cell analysis; t-SNE.

Figure 1

Features of dimensionally reduced data are sensitive to the choice of algorithm and algorithmic settings (A and C) Four dimensionally reduced representations of RNA-seq measurements from 4,771 bone marrow cells collected by the Tabula Muris Consortium generated by t-SNE at $k_{\text{Eff}}$$\approx 15$ (A) and 150 (C) (perplexity = 10 and 120, respectively; custom variation of the openTSNE implementation^51,52) and by UMAP at n_neighbors = k = 15 and 150. Ten previously annotated cell types provided by Schaum et al. are colored and labeled. The same cells are colored and labeled in each panel. (B) The number of nearest neighbors, k, is set to its default value, 15, in UMAP. Following the method in supplemental section S3, we use t-SNE with a similar number of nearest neighbors (k_Eff = 15) in (A). (C and D) We visualize the data using t-SNE and UMAP, respectively, at a much larger number of nearest neighbors: k_Eff ≈ 150 in (C) and k = 150 in (D).

Figure 2

A schematic of the EMBEDR algorithm (A) The data (5,037 FACS-sorted marrow cells from Schaum et al. shown as a heatmap) are embedded in 2D using a DR method several times (here: UMAP with k = n_neighbors = 100). For each sample, the distances to neighboring samples are calculated in both the original data, ${\overrightarrow{x}}_i\to {\overrightarrow{x}}_j$, and the low-dimensional embedding, ${\overrightarrow{y}}_i\to {\overrightarrow{y}}_j$. An example cell is illustrated by a red star in each of the embeddings. These distance distributions are compared to calculate $EE{S}_{i,n}$, a quality score for each cell in each embedding. (B) The same procedure as in (A) is conducted using null datasets constructed via marginal resampling (see Figure 3). A purple star indicates a sample point in each null embedding. (C) The individual $EE{S}_{i,n}$ values are compared with the null distribution of $EE{S}^{\text{∗}}$ to estimate a p value for each cell’s embedding quality. This p value corresponds to the empirical likelihood that the null data could generate an observed or better embedding quality. (D) The UMAP embedding of the data from (A) is shown. Cells in this embedding are colored according to the p values calculated in (C) , so that embedding quality can easily be visualized across an embedding. The light purple cells are those whose neighborhoods are better preserved than expected by random chance.

Figure 3

An overview of marginal resampling for generating null datasets (A) Gene expression data for real and resampled scRNA-seq data (FACS-sorted marrow cells⁸) are shown as heatmaps. (B) The first and second principal component of the data in (A) are plotted against each other, and the corresponding marginal distributions are shown to the top and right. Kernel density estimates are also plotted on the marginal distributions. (C and D) The effect of marginal resampling to generate null distributions is shown, where the data and a null dataset are embedded using UMAP at k = 15 and t-SNE at $k_{\text{Eff}}\approx 60$, respectively, which correspond to the default parameters for those algorithms.

Figure 4

Optimizing DR algorithm hyperparameters generates high-quality embeddings A total of 4,771 bone marrow cells from several mice were embedded with t-SNE 5 times at several values of $k_{\text{Eff}}$ and the EMBEDR p value was calculated using 10 null embeddings. (A–C) Embeddings generated at three interesting values of $k_{\text{Eff}}$; each cell is colored by the EMBEDR p value (shown by the color bar) in (D). In (A), $k_{\text{Eff}}\approx 40$ corresponds to the default t-SNE parameter (perplexity = 30) in most implementations of t-SNE.^, (C) An embedding generated using $k_{\text{Eff}}\approx 1200$ (perplexity = 1,000), which corresponds to the largest fraction of cells being well represented in the lower-dimensional embedding. Similarly, (B) shows the results at $k_{\text{Eff}}\approx 150$ (perplexity = 100), which corresponds to a second, smaller minimum in the p values. (D) The distributions of p values are shown as box-and-whisker plots over each value of $k_{\text{Eff}}$ and the median of the boxplot at $k_{\text{Eff}}\approx$ 1,200 indicates that a substantial fraction of cells are best embedded at that hyperparameter value.

Figure 5

EMBEDR facilitates direct comparisons of DR methods (A–D) A total of 4,711 cells from the Tabula Muris marrow tissue are embedded by t-SNE and UMAP at default (A and B) and EMBEDR-optimized (C and D) numbers of nearest neighbors. Each cell in each embedding is colored by the EMBEDR p value according to the color bars on the right. The p values are calculated as in Figure 2 and in supplemental Section S4 using $N_{\text{embed}}=25$ applications of t-SNE/UMAP to the data and $N_{\text{embed}}^{\text{∗}}=10$ embeddings of null data. In the boxes below each panel, the number (percentage) of cells at each p value threshold are shown (indicated by the corresponding color), with the threshold containing a plurality of cells shown in bold.

Figure 6

Different cell types are best embedded at a variety of scales Using annotations from the Tabula Muris project, the embedding quality of different cell types in the Marrow data can be examined individually across values of k_Eff. (A) Six identified cell types from the bone marrow tissue are shown, where each cell with a given annotation is shown as an individual line. The colored boxes indicate the median p value across all cells with that annotation, and the solid lines indicate the 90th percentiles. Similar plots for all cell types are shown in Figure S13. Embeddings at k_Eff ≈ 150 and 1,200 are shown in (B and C), respectively. (B and C) The cells corresponding to each cell type are highlighted with the same color as in (A). Cells with an EMBEDR p value below ${10}^{-3}$ (the gray line in A) are opaque, while other cells with a highlighted annotation are lightly shaded. The fractions of such cells in an annotation are shown in the colored boxes below the embeddings. Other cell types are shown in gray for context.

Figure 7

A cell-wise optimized embedding reveals clear biological signals Adapting t-SNE to use a different scale for each sample in the Tabula Muris marrow data generates a well-structured representation of the data. (A) The unlabeled embedding is presented. (B and C) To generate this embedding, the scale at which a cell’s p value was minimized was used to set k_Eff for that cell. This k_Eff is shown in (B) and the minimal p value achieved by a cell across the sweep is shown in (C). (D) Applying DBSCAN with eps set based on the pairwise distance (PWD) distribution of cells in the embedding (specifically, the 1.5th percentile of PWDs) detected the seven indicated clusters. Any Tabula Muris cell-type annotation for which more than 20 cells overlapped with a DBSCAN label was given a different shade of the cluster color. (E) These cell annotations and colors are shown as a confusion table.