Systematic evaluation of differential splicing tools for RNA-seq studies

Abstract

Differential splicing (DS) is a post-transcriptional biological process with critical, wide-ranging effects on a plethora of cellular activities and disease processes. To date, a number of computational approaches have been developed to identify and quantify differentially spliced genes from RNA-seq data, but a comprehensive intercomparison and appraisal of these approaches is currently lacking. In this study, we systematically evaluated 10 DS analysis tools for consistency and reproducibility, precision, recall and false discovery rate, agreement upon reported differentially spliced genes and functional enrichment. The tools were selected to represent the three different methodological categories: exon-based (DEXSeq, edgeR, JunctionSeq, limma), isoform-based (cuffdiff2, DiffSplice) and event-based methods (dSpliceType, MAJIQ, rMATS, SUPPA). Overall, all the exon-based methods and two event-based methods (MAJIQ and rMATS) scored well on the selected measures. Of the 10 tools tested, the exon-based methods performed generally better than the isoform-based and event-based methods. However, overall, the different data analysis tools performed strikingly differently across different data sets or numbers of samples.

Keywords: RNA-seq; differential splicing; event-based methods; exon-based methods; isoform-based methods; splicing events.

Figure 1

Schematic illustration of the comparison of DS tools. In total, 10 different tools were assessed in four different RNA-seq data sets.

Figure 2

Median and standard error of the number of detections, precision, recall and FDR of the 10 compared methods in PCa and HCa data sets with different numbers of replicates. Number of DS genes in the (A) PCa and (B) HCa data set. Precision in the (C) PCa and (D) HCa data set. Recall in the (E) PCa and (F) HCa data set. FDR in the (G) PCa and (H) HCa data set. The points and error bars correspond to the median and the standard error of the 10 randomly sampled subsets for the different numbers of replicates.

Figure 3

Similarity between the methods in the complete PCa and HCa data sets. Overlap of top 500 ranked DS genes between the methods in the (A) PCa and (B) HCa data set. Genes were ranked based on the FDR in all methods except for DiffSplice, which provided its own test statistic to rank the genes instead. dSpliceType was excluded from this comparison as it allowed listing genes only until FDR of 0.05 which provided less than 500 genes in both data sets.

Figure 4

Heatmap of the P-values of the top enriched GO biological processes across the methods in the complete (A) PCa and (B) HCa data sets. Grey colour represents missing values.

Figure 5

Memory usage and run time of the methods with different numbers of replicates in HCa data set. (A) Run time and (B) maximum memory required, as measured by Linux process accounting tool acct. Values are on log10 scale.