A Framework for Comparison and Assessment of Synthetic RNA-Seq Data

Abstract

The ever-growing number of methods for the generation of synthetic bulk and single cell RNA-seq data have multiple and diverse applications. They are often aimed at benchmarking bioinformatics algorithms for purposes such as sample classification, differential expression analysis, correlation and network studies and the optimization of data integration and normalization techniques. Here, we propose a general framework to compare synthetically generated RNA-seq data and select a data-generating tool that is suitable for a set of specific study goals. As there are multiple methods for synthetic RNA-seq data generation, researchers can use the proposed framework to make an informed choice of an RNA-seq data simulation algorithm and software that are best suited for their specific scientific questions of interest.

Keywords: RNA-seq; comparative study; differential expression; sample classification; simulated data.

Figure 1

Methodological framework for synthetic RNA-seq data generation for benchmarking of algorithms for statistical and pattern recognition analyses.

Figure 2

Application of the framework for comparison of bulk RNA-seq generators. Different colors indicate different types of methods. compcodeR and powsimR are parametric, SPsimSeq is semiparametric, and SimSeq and seqgendiff are nonparametric.

Figure 3

Q-Q plots of five synthetic data samples generated by the respective data generators (y axis) vs. five samples used as input for those data generators (x-axis), with (A–E) NGSSPPG1 samples used as input data; (F–J) NGSSPP2 samples as input data; (K–O) AD samples as input data. The axes represent the quantiles of the respective distributions. Blue: linear regression line. Red: diagonal line.

Figure 4

Dispersion vs. BCV plots of the: (A) NGSSPPG1; (B) NGSSPPG2; and (C) AD datasets and the datasets generated with these as input. Black dots: gene-wise dispersion estimates. Red curve: fitted mean-dispersion relationship. Blue circles: final dispersion estimates.

Figure 5

Mean-variance plots of the: (A) NGSSPPG1; (B) NGSSPPG2; and (C) AD datasets and the datasets generated with these as input.

Figure 6

Feature-feature correlation plots of the (A) NGSSPPG1; (B) NGSSPPG2; and (C) AD data and the datasets generated with these as input.

Figure 7

Volcano plots for: (A) NGSSPPG1; (B) NGSSPPG2; and (C) AD datasets and the datasets generated with these as input. Blue dots: transcripts with p-value > 0.05, denoting significant differential expression. Red dots: transcripts with p-value > 0.05 and $lo{g}_2\left(FoldChange\right)$ > 1 or with p-value > 0.05 and $lo{g}_2\left(FoldChange\right)$ < −1.

Figure 8

PCA plots for: (A) NGSSPPG1; (B) NGSSPPG2; and (C) AD datasets and the datasets generated with these as input.