Comparisons of performances of structural variants detection algorithms in solitary or combination strategy

Abstract

Structural variants (SVs) have been associated with changes in gene expression, which may contribute to alterations in phenotypes and disease development. However, the precise identification and characterization of SVs remain challenging. While long-read sequencing offers superior accuracy for SV detection, short-read sequencing remains essential due to practical and cost considerations, as well as the need to analyze existing short-read datasets. Numerous algorithms for short-read SV detection exist, but none are universally optimal, each having limitations for specific SV sizes and types. In this study, we evaluated the efficacy of six advanced SV detection algorithms, including the commercial software DRAGEN, using the GIAB v0.6 Tier 1 benchmark and HGSVC2 cell lines. We employed both individual and combination strategies, with systematic assessments of recall, precision, and F1 scores. Our results demonstrate that the union combination approach enhanced detection capabilities, surpassing single algorithms in identifying deletions and insertions, and delivered comparable recall and F1 scores to the commercial software DRAGEN. Interestingly, expanding the number of algorithms from three to five in the combination did not enhance performance, highlighting the efficiency of a well-chosen ensemble over a larger algorithmic pool.

Fig 1. SV detection workflow.

SV detection workflow starts with benchmark set collection from GIAB and HGSVC2, followed by sequence alignment using BWA-MEM to produce BAM files. These BAM files are then processed by various SV callers (DRAGEN, Manta, DELLY, LUMPY, GRIDSS, SvABA) to generate VCF files, which are subsequently used for performance assessment, including single tool and combination strategies, based on recall, precision, and F1 score.

Fig 2. The SVs detected by six SV callers.

(A) SVs of sizes ≥ 50 bp and CTX detected by individual SV callers in samples, HG002, HG00514, HG00733, and NA19240. (B) Length distribution of different variants for all samples detected by individual SV callers. The maximum, minimum, and median are based on the integrated values from all sample sets. DEL: deletion, INS: insertion, DUP: duplication, INV: inversion, and CTX: complex translocation.

Fig 3. The distribution of SVs in truth sets and the performance of individual algorithms.

(A) The distribution of SVs size and types in truth sets. (B) The comparison of False negative (FN) and False positive (FP) numbers among individual algorithms. (C) The precision, recall, and F1 score of each individual algorithm in detecting “DELs” and “INSs” with the size ≥ 50 bp. DEL: deletion, INS: insertion.

Fig 4. Concordance of neighbor SVs detected by member callers in different combination strategies.

(A) The agreement among three SV detection tools (Manta, DELLY, and GRIDSS). (B) The agreement among five SV detection tools (Manta, DELLY, GRIDSS, LUMPY, and SvABA).

Fig 5. Distribution and performance of combination strategies.

(A) The distribution of SVs in different combination strategies. (B) The comparison of False negative (FN) and False positive (FP) numbers among individual algorithms. (C) The precision, recall, and F1 score of combination strategies in the detection of DELs and INSs. DEL: deletion, INS: insertions, DUP: duplication, INV: inversion.

Fig 6. Combination performance of each single caller and combination strategy in detecting DELs and INSs.

The maximum, minimum, and macro average of recall, precision, and F1 score are based on the integrated values from all sample sets.