SMaSH: a benchmarking toolkit for human genome variant calling

Abstract

Motivation: Computational methods are essential to extract actionable information from raw sequencing data, and to thus fulfill the promise of next-generation sequencing technology. Unfortunately, computational tools developed to call variants from human sequencing data disagree on many of their predictions, and current methods to evaluate accuracy and computational performance are ad hoc and incomplete. Agreement on benchmarking variant calling methods would stimulate development of genomic processing tools and facilitate communication among researchers.

Results: We propose SMaSH, a benchmarking methodology for evaluating germline variant calling algorithms. We generate synthetic datasets, organize and interpret a wide range of existing benchmarking data for real genomes and propose a set of accuracy and computational performance metrics for evaluating variant calling methods on these benchmarking data. Moreover, we illustrate the utility of SMaSH to evaluate the performance of some leading single-nucleotide polymorphism, indel and structural variant calling algorithms.

Availability and implementation: We provide free and open access online to the SMaSH tool kit, along with detailed documentation, at smash.cs.berkeley.edu

Fig. 1.

An ‘ideal’ benchmarking dataset satisfies three properties: it contains real reads (R), it includes comprehensive validation of the underlying genome (C) and its underlying genome is human (H). SMaSH contains three types of benchmarking datasets, each of which satisfies two of the three desirable properties of an ideal dataset, so as to cover all three properties

Fig. 2.

Schematic illustrating process by which SMaSH’s first ‘Mouse’ dataset is generated. (a) Our ideal setup in which the B6 strain (with comprehensive validation and corresponding short reads) serves as the sample and the DBA strain serves as the reference. (b) Publicly available data (note that the B6 validation data are the canonical mouse reference). (c) Construction of an approximate DBA validation set (the ‘fake’ reference) by leveraging a rough set of variants for the DBA strain called relative to the canonical reference