Extend the benchmarking indel set by manual review using the individual cell line sequencing data from the Sequencing Quality Control 2 (SEQC2) project

Abstract

Accurate indel calling plays an important role in precision medicine. A benchmarking indel set is essential for thoroughly evaluating the indel calling performance of bioinformatics pipelines. A reference sample with a set of known-positive variants was developed in the FDA-led Sequencing Quality Control Phase 2 (SEQC2) project, but the known indels in the known-positive set were limited. This project sought to provide an enriched set of known indels that would be more translationally relevant by focusing on additional cancer related regions. A thorough manual review process completed by 42 reviewers, two advisors, and a judging panel of three researchers significantly enriched the known indel set by an additional 516 indels. The extended benchmarking indel set has a large range of variant allele frequencies (VAFs), with 87% of them having a VAF below 20% in reference Sample A. The reference Sample A and the indel set can be used for comprehensive benchmarking of indel calling across a wider range of VAF values in the lower range. Indel length was also variable, but the majority were under 10 base pairs (bps). Most of the indels were within coding regions, with the remainder in the gene regulatory regions. Although high confidence can be derived from the robust study design and meticulous human review, this extensive indel set has not undergone orthogonal validation. The extended benchmarking indel set, along with the indels in the previously published known-positive set, was the truth set used to benchmark indel calling pipelines in a community challenge hosted on the precisionFDA platform. This benchmarking indel set and reference samples can be utilized for a comprehensive evaluation of indel calling pipelines. Additionally, the insights and solutions obtained during the manual review process can aid in improving the performance of these pipelines.

Keywords: Benchmarking; Bioinformatics; Indel; Precision medicine; Quality control.

Figure 1

(A) Reference Sample A was created by mixing equal mass of DNA samples from 10 cancer cell lines, i.e., Myeloma (B-lymphocyte, BLY), Glioblastoma (brain, BRA), Adenocarcinoma (breast, BRE), Adenocarcinoma (cervix, CRV), Liposarcoma (soft tissue, LIP), Hepatoblastoma (liver, LIV), Lymphoma (macrophage, MAC), Melanoma (skin, SKN), Carcinoma (testes, TES), Carcinoma (T-lymphoblast, TLY). Sample B was DNA sample derived from a normal male control cell line. These 11 DNA samples were sequenced using three WES panels, i.e., Roche MedExome panel (WES1), IDT xGen Exome panel (WES2), and Agilent SureSelect Exome panel (WES3). For each WES panel, two library replicates were made and sequenced. In total, 66 BAM files were obtained after alignment to hg19 reference genome. (B) Overall block diagram of the manual review process in this study. The figure illustrates the main steps of the indel manual review process.

Figure 2

Indel VAF distribution. (A) Binned VAF distribution by percentage of indels. Each color represents a different VAF bin. (B) Number of indel variants found across the VAF spectrum of VAF. Bar height represents more the number of indel variants found within each VAF bin. VAF bins are represented on the X-axis and are non-linear.

Figure 3

Length of insertions (A) and deletions (B). Bar height represents the number of indel variants found at in each that length. The x-axis represents the length of the genomic event, either insertion (A) or deletion (B).

Figure 4

Genomic impact of indels, broken down into frameshift, UTR, in-frame, intron/intergenic region, and others.