Benchmarking and improving the performance of variant-calling pipelines with RecallME

Abstract

Motivation: The steady increment of Whole Genome/Exome sequencing and the development of novel Next Generation Sequencing-based gene panels requires continuous testing and validation of variant calling (VC) pipelines and the detection of sequencing-related issues to be maintained up-to-date and feasible for the clinical settings. State of the art tools are reliable when used to compute standard performance metrics. However, the need for an automated software to discriminate between bioinformatic and sequencing issues and to optimize VC parameters remains unmet.

Results: The aim of the current work is to present RecallME, a bioinformatic suite that tracks down difficult-to-detect variants as insertions and deletions in highly repetitive regions, thus providing the maximum reachable recall for both single nucleotide variants and small insertion and deletions and to precisely guide the user in the pipeline optimization process.

Availability and implementation: Source code is freely available under MIT license at https://github.com/mazzalab-ieo/recallme. RecallME web application is available at https://translational-oncology-lab.shinyapps.io/recallme/. To use RecallME, users must obtain a license for ANNOVAR by themselves.

Figure 1.

RecallME allows the maximization of the recall in reference samples. (A) Flowchart that shows how RecallME suite works with and without the bam file as input. First, bcftools norm splits the multi-allelic variants, then the query and the ground truth VCF files are converted in annovar inputs to harmonize variant notations. Bedtools genomecov function computes the number of bps that are not considered as high confidence by subtracting bps outside the bed file (to compute the number of true negatives and, consequently, the specificity). Bam-readcount look for SNVs and INDELs within the bam file to check if the recall can be maximized. The recaller.R and the pileup_recaller.R scripts compute the standard metrics as recall, precision, specificity, F1-score, FDR, and the Recall Max. (B) Barplots showing recall metrics in TVC and LoFreq pipelines (ION technology) and GATK-HC (Illumina-based) computed by som.py and RecallME (before and after BAM file re-check step) in SNV and INDEL calls in NA12878 sample. (C) Barplots showing recall metrics in GATK-HC and Pepper DV pipelines (Illumina and ONT-based) computed by som.py and RecallME (before and after BAM file re-check step) in SNV and INDEL calls in HD793 sample (Illumina and ONT). (D) Barplots showing accuracy metrics in Mutect2 pipeline computed by som.py and RecallME (before and after BAM file re-check step, i.e. RecallMax) in SNV and INDEL calls in SEQC2 somatic dataset. (E) Barplots showing recall metrics in TVC pipeline (ION technology) computed by hap.py and RecallME (before and after BAM file re-check step) in SNV and INDEL calls in NA12878 dataset. (F) TVC parameters distributions across TPs, FPs and b-FNs in SNV variants. Statistically significant differences have been found in VAF, DP, and STB (Mann–Whitney two-sided test). (G) TVC parameters distributions across TPs, FPs, and b-FNs in INDEL variants. Statistically significant differences have been found in VAF, DP, and STB (Mann–Whitney two-sided test). (H) Accuracies in TVC performances (NA12878) by tuning optimal cutpoints for VAF, DP, and STB for SNVs, INDELs and whole calls (WHOLE).