. 2021 Apr 16;22(1):111.

doi: 10.1186/s13059-021-02316-z.

A verified genomic reference sample for assessing performance of cancer panels detecting small variants of low allele frequency

Wendell Jones¹, Binsheng Gong², Natalia Novoradovskaya³, Dan Li², Rebecca Kusko⁴, Todd A Richmond⁵, Donald J Johann Jr⁶, Halil Bisgin⁷, Sayed Mohammad Ebrahim Sahraeian⁸, Pierre R Bushel⁹, Mehdi Pirooznia¹⁰, Katherine Wilkins¹¹, Marco Chierici¹², Wenjun Bao¹³, Lee Scott Basehore³, Anne Bergstrom Lucas¹¹, Daniel Burgess¹⁴, Daniel J Butler¹⁵, Simon Cawley¹⁶, Chia-Jung Chang¹⁷, Guangchun Chen¹⁸, Tao Chen¹⁹, Yun-Ching Chen¹⁰, Daniel J Craig²⁰, Angela Del Pozo²¹, Jonathan Foox¹⁵, Margherita Francescatto¹², Yutao Fu²², Cesare Furlanello¹², Kristina Giorda²³, Kira P Grist²⁴, Meijian Guan¹³, Yingyi Hao²⁵, Scott Happe²⁶, Gunjan Hariani²⁴, Nathan Haseley²⁷, Jeff Jasper²⁴, Giuseppe Jurman¹², David Philip Kreil²⁸, Paweł Łabaj^{29

30}, Kevin Lai³¹, Jianying Li³², Quan-Zhen Li¹⁸, Yulong Li³³, Zhiguang Li³³, Zhichao Liu², Mario Solís López^{21

34}, Kelci Miclaus¹³, Raymond Miller¹¹, Vinay K Mittal²², Marghoob Mohiyuddin⁸, Carlos Pabón-Peña¹¹, Barbara L Parsons³⁵, Fujun Qiu³⁶, Andreas Scherer^{34

37}, Tieliu Shi³⁸, Suzy Stiegelmeyer³⁹, Chen Suo⁴⁰, Nikola Tom^{34

41}, Dong Wang², Zhining Wen²⁵, Leihong Wu², Wenzhong Xiao^{17

42}, Chang Xu⁴³, Ying Yu⁴⁴, Jiyang Zhang⁴⁴, Yifan Zhang⁴⁵, Zhihong Zhang³⁶, Yuanting Zheng^{2

44}, Christopher E Mason¹⁵, James C Willey⁴⁶, Weida Tong², Leming Shi^{44

47

48}, Joshua Xu⁴⁹

Affiliations

¹ Q2 Solutions - EA Genomics, 5927 S Miami Blvd., Morrisville, NC, 27560, USA. Wendell.Jones@q2labsolutions.com.
² Division of Bioinformatics and Biostatistics, National Center for Toxicological Research, US Food and Drug Administration, Jefferson, AR, 72079, USA.
³ Agilent Technologies, 11011 N Torrey Pines Rd., La Jolla, CA, 92037, USA.
⁴ Immuneering Corporation, One Broadway, 14th Floor, Cambridge, MA, 02142, USA.
⁵ Market & Application Development Bioinformatics, Roche Sequencing Solutions Inc., 4300 Hacienda Dr., Pleasanton, CA, 94588, USA.
⁶ Winthrop P Rockefeller Cancer Institute, University of Arkansas for Medical Sciences, 4301 W Markham St., Little Rock, AR, 72205, USA.
⁷ Department of Computer Science, Engineering and Physics, University of Michigan-Flint, Flint, MI, 48502, USA.
⁸ Bioinformatics Research & Early Development, Roche Sequencing Solutions Inc., 1301 Shoreway Rd., Suite 7 #300, Belmont, CA, 94002, USA.
⁹ National Institute of Environmental Health Sciences, Research Triangle Park, Durham, NC, 27709, USA.
¹⁰ Bioinformatics and Computational Biology Laboratory, National Heart Lung and Blood Institute, National Institutes of Health, Bethesda, MD, 20892, USA.
¹¹ Agilent Technologies, 5301 Stevens Creek Blvd., Santa Clara, CA, 95051, USA.
¹² Fondazione Bruno Kessler, 38123, Trento, Italy.
¹³ JMP Life Sciences, SAS Institute Inc., Cary, NC, 27519, USA.
¹⁴ (formerly) Research and Development, Roche Sequencing Solutions Inc., 500 South Rosa Rd., Madison, WI, 53719, USA.
¹⁵ Department of Physiology and Biophysics, Weill Cornell Medicine, Cornell University, New York, NY, 10065, USA.
¹⁶ (formerly) Clinical Sequencing Division, Thermo Fisher Scientific, 180 Oyster Point Blvd., South San Francisco, CA, 94080, USA.
¹⁷ Stanford Genome Technology Center, Stanford University, Palo Alto, CA, 94304, USA.
¹⁸ Department of Immunology, Genomics and Microarray Core Facility, University of Texas Southwestern Medical Center, 5323 Harry Hine Blvd., Dallas, TX, 75390, USA.
¹⁹ University of Texas Southwestern Medical Center, 2330 Inwood Rd., Dallas, TX, 75390, USA.
²⁰ Department of Medicine, College of Medicine and Life Sciences, The University of Toledo, Toledo, OH, 43614, USA.
²¹ Institute of Medical and Molecular Genetics (INGEMM), Hospital Universitario La Paz, CIBERER Instituto de Salud Carlos III, 28046, Madrid, Spain.
²² Thermo Fisher Scientific, 110 Miller Ave., Ann Arbor, MI, 48104, USA.
²³ Marketing, Integrated DNA Technologies, Inc., 1710 Commercial Park, Coralville, IA, 52241, USA.
²⁴ Q2 Solutions - EA Genomics, 5927 S Miami Blvd., Morrisville, NC, 27560, USA.
²⁵ College of Chemistry, Sichuan University, Chengdu, 610064, Sichuan, China.
²⁶ Agilent Technologies, 1834 State Hwy 71 West, Cedar Creek, TX, 78612, USA.
²⁷ Illumina Inc., 5200 Illumina Way, San Diego, CA, 92122, USA.
²⁸ Bioinformatics Research, Institute of Molecular Biotechnology, Boku University Vienna, Vienna, Austria.
²⁹ Małopolska Centre of Biotechnology, Jagiellonian University, Krakow, Poland.
³⁰ Department of Biotechnology, Boku University, Vienna, Austria.
³¹ Bioinformatics, Integrated DNA Technologies, Inc., 1710 Commercial Park, Coralville, IA, 52241, USA.
³² Kelly Government Solutions, Inc., Research Triangle Park, NC, 27709, USA.
³³ Center of Genome and Personalized Medicine, Institute of Cancer Stem Cell, Dalian Medical University, Dalian, Liaoning, China.
³⁴ EATRIS ERIC- European Infrastructure for Translational Medicine, De Boelelaan 1118, 1081, HZ, Amsterdam, The Netherlands.
³⁵ Division of Genetic and Molecular Toxicology, National Center for Toxicological Research, US Food and Drug Administration, Jefferson, AR, 72079, USA.
³⁶ Research and Development, Burning Rock Biotech, Shanghai, 201114, China.
³⁷ Institute for Molecular Medicine Finland (FIMM), Nordic EMBL Partnership for Molecular Medicine, HiLIFE Unit, Biomedicum Helsinki 2U (D302b), FI-00014 University of Helsinki, P.O. Box 20 (Tukholmankatu 8), Helsinki, Finland.
³⁸ Center for Bioinformatics and Computational Biology, and the Institute of Biomedical Sciences, School of Life Sciences, East China Normal University, 500 Dongchuan Rd, Shanghai, 200241, China.
³⁹ University of North Carolina Health, 101 Manning Drive, Chapel Hill, NC, 27514, USA.
⁴⁰ Department of Epidemiology, School of Public Health, Fudan University, Shanghai, China.
⁴¹ Center of Molecular Medicine, Central European Institute of Technology, Masaryk University, Kamenice 5, 625 00, Brno, Czech Republic.
⁴² Massachusetts General Hospital, Harvard Medical School, Boston, MA, 02114, USA.
⁴³ Research and Development, QIAGEN Sciences Inc., Frederick, MD, 21703, USA.
⁴⁴ State Key Laboratory of Genetic Engineering, School of Life Sciences and Shanghai Cancer Hospital/Cancer Institute, Fudan University, Shanghai, 200438, China.
⁴⁵ University of Arkansas at Little Rock, Little Rock, AR, 72204, USA.
⁴⁶ Departments of Medicine, Pathology, and Cancer Biology, College of Medicine and Life Sciences, University of Toledo Health Sciences Campus, 3000 Arlington Ave, Toledo, OH, 43614, USA.
⁴⁷ Human Phenome Institute, Fudan University, Shanghai, 201203, China.
⁴⁸ Fudan-Gospel Joint Research Center for Precision Medicine, Fudan University, Shanghai, 200438, China.
⁴⁹ Division of Bioinformatics and Biostatistics, National Center for Toxicological Research, US Food and Drug Administration, Jefferson, AR, 72079, USA. Joshua.xu@fda.hhs.gov.

PMID: 33863366
PMCID: PMC8051128
DOI: 10.1186/s13059-021-02316-z

A verified genomic reference sample for assessing performance of cancer panels detecting small variants of low allele frequency

Wendell Jones et al. Genome Biol. 2021.

. 2021 Apr 16;22(1):111.

doi: 10.1186/s13059-021-02316-z.

Authors

Affiliations

¹ Q2 Solutions - EA Genomics, 5927 S Miami Blvd., Morrisville, NC, 27560, USA. Wendell.Jones@q2labsolutions.com.
² Division of Bioinformatics and Biostatistics, National Center for Toxicological Research, US Food and Drug Administration, Jefferson, AR, 72079, USA.
³ Agilent Technologies, 11011 N Torrey Pines Rd., La Jolla, CA, 92037, USA.
⁴ Immuneering Corporation, One Broadway, 14th Floor, Cambridge, MA, 02142, USA.
⁵ Market & Application Development Bioinformatics, Roche Sequencing Solutions Inc., 4300 Hacienda Dr., Pleasanton, CA, 94588, USA.
⁶ Winthrop P Rockefeller Cancer Institute, University of Arkansas for Medical Sciences, 4301 W Markham St., Little Rock, AR, 72205, USA.
⁷ Department of Computer Science, Engineering and Physics, University of Michigan-Flint, Flint, MI, 48502, USA.
⁸ Bioinformatics Research & Early Development, Roche Sequencing Solutions Inc., 1301 Shoreway Rd., Suite 7 #300, Belmont, CA, 94002, USA.
⁹ National Institute of Environmental Health Sciences, Research Triangle Park, Durham, NC, 27709, USA.
¹⁰ Bioinformatics and Computational Biology Laboratory, National Heart Lung and Blood Institute, National Institutes of Health, Bethesda, MD, 20892, USA.
¹¹ Agilent Technologies, 5301 Stevens Creek Blvd., Santa Clara, CA, 95051, USA.
¹² Fondazione Bruno Kessler, 38123, Trento, Italy.
¹³ JMP Life Sciences, SAS Institute Inc., Cary, NC, 27519, USA.
¹⁴ (formerly) Research and Development, Roche Sequencing Solutions Inc., 500 South Rosa Rd., Madison, WI, 53719, USA.
¹⁵ Department of Physiology and Biophysics, Weill Cornell Medicine, Cornell University, New York, NY, 10065, USA.
¹⁶ (formerly) Clinical Sequencing Division, Thermo Fisher Scientific, 180 Oyster Point Blvd., South San Francisco, CA, 94080, USA.
¹⁷ Stanford Genome Technology Center, Stanford University, Palo Alto, CA, 94304, USA.
¹⁸ Department of Immunology, Genomics and Microarray Core Facility, University of Texas Southwestern Medical Center, 5323 Harry Hine Blvd., Dallas, TX, 75390, USA.
¹⁹ University of Texas Southwestern Medical Center, 2330 Inwood Rd., Dallas, TX, 75390, USA.
²⁰ Department of Medicine, College of Medicine and Life Sciences, The University of Toledo, Toledo, OH, 43614, USA.
²¹ Institute of Medical and Molecular Genetics (INGEMM), Hospital Universitario La Paz, CIBERER Instituto de Salud Carlos III, 28046, Madrid, Spain.
²² Thermo Fisher Scientific, 110 Miller Ave., Ann Arbor, MI, 48104, USA.
²³ Marketing, Integrated DNA Technologies, Inc., 1710 Commercial Park, Coralville, IA, 52241, USA.
²⁴ Q2 Solutions - EA Genomics, 5927 S Miami Blvd., Morrisville, NC, 27560, USA.
²⁵ College of Chemistry, Sichuan University, Chengdu, 610064, Sichuan, China.
²⁶ Agilent Technologies, 1834 State Hwy 71 West, Cedar Creek, TX, 78612, USA.
²⁷ Illumina Inc., 5200 Illumina Way, San Diego, CA, 92122, USA.
²⁸ Bioinformatics Research, Institute of Molecular Biotechnology, Boku University Vienna, Vienna, Austria.
²⁹ Małopolska Centre of Biotechnology, Jagiellonian University, Krakow, Poland.
³⁰ Department of Biotechnology, Boku University, Vienna, Austria.
³¹ Bioinformatics, Integrated DNA Technologies, Inc., 1710 Commercial Park, Coralville, IA, 52241, USA.
³² Kelly Government Solutions, Inc., Research Triangle Park, NC, 27709, USA.
³³ Center of Genome and Personalized Medicine, Institute of Cancer Stem Cell, Dalian Medical University, Dalian, Liaoning, China.
³⁴ EATRIS ERIC- European Infrastructure for Translational Medicine, De Boelelaan 1118, 1081, HZ, Amsterdam, The Netherlands.
³⁵ Division of Genetic and Molecular Toxicology, National Center for Toxicological Research, US Food and Drug Administration, Jefferson, AR, 72079, USA.
³⁶ Research and Development, Burning Rock Biotech, Shanghai, 201114, China.
³⁷ Institute for Molecular Medicine Finland (FIMM), Nordic EMBL Partnership for Molecular Medicine, HiLIFE Unit, Biomedicum Helsinki 2U (D302b), FI-00014 University of Helsinki, P.O. Box 20 (Tukholmankatu 8), Helsinki, Finland.
³⁸ Center for Bioinformatics and Computational Biology, and the Institute of Biomedical Sciences, School of Life Sciences, East China Normal University, 500 Dongchuan Rd, Shanghai, 200241, China.
³⁹ University of North Carolina Health, 101 Manning Drive, Chapel Hill, NC, 27514, USA.
⁴⁰ Department of Epidemiology, School of Public Health, Fudan University, Shanghai, China.
⁴¹ Center of Molecular Medicine, Central European Institute of Technology, Masaryk University, Kamenice 5, 625 00, Brno, Czech Republic.
⁴² Massachusetts General Hospital, Harvard Medical School, Boston, MA, 02114, USA.
⁴³ Research and Development, QIAGEN Sciences Inc., Frederick, MD, 21703, USA.
⁴⁴ State Key Laboratory of Genetic Engineering, School of Life Sciences and Shanghai Cancer Hospital/Cancer Institute, Fudan University, Shanghai, 200438, China.
⁴⁵ University of Arkansas at Little Rock, Little Rock, AR, 72204, USA.
⁴⁶ Departments of Medicine, Pathology, and Cancer Biology, College of Medicine and Life Sciences, University of Toledo Health Sciences Campus, 3000 Arlington Ave, Toledo, OH, 43614, USA.
⁴⁷ Human Phenome Institute, Fudan University, Shanghai, 201203, China.
⁴⁸ Fudan-Gospel Joint Research Center for Precision Medicine, Fudan University, Shanghai, 200438, China.
⁴⁹ Division of Bioinformatics and Biostatistics, National Center for Toxicological Research, US Food and Drug Administration, Jefferson, AR, 72079, USA. Joshua.xu@fda.hhs.gov.

PMID: 33863366
PMCID: PMC8051128
DOI: 10.1186/s13059-021-02316-z

Abstract

Background: Oncopanel genomic testing, which identifies important somatic variants, is increasingly common in medical practice and especially in clinical trials. Currently, there is a paucity of reliable genomic reference samples having a suitably large number of pre-identified variants for properly assessing oncopanel assay analytical quality and performance. The FDA-led Sequencing and Quality Control Phase 2 (SEQC2) consortium analyze ten diverse cancer cell lines individually and their pool, termed Sample A, to develop a reference sample with suitably large numbers of coding positions with known (variant) positives and negatives for properly evaluating oncopanel analytical performance.

Results: In reference Sample A, we identify more than 40,000 variants down to 1% allele frequency with more than 25,000 variants having less than 20% allele frequency with 1653 variants in COSMIC-related genes. This is 5-100× more than existing commercially available samples. We also identify an unprecedented number of negative positions in coding regions, allowing statistical rigor in assessing limit-of-detection, sensitivity, and precision. Over 300 loci are randomly selected and independently verified via droplet digital PCR with 100% concordance. Agilent normal reference Sample B can be admixed with Sample A to create new samples with a similar number of known variants at much lower allele frequency than what exists in Sample A natively, including known variants having allele frequency of 0.02%, a range suitable for assessing liquid biopsy panels.

Conclusion: These new reference samples and their admixtures provide superior capability for performing oncopanel quality control, analytical accuracy, and validation for small to large oncopanels and liquid biopsy assays.

PubMed Disclaimer

Conflict of interest statement

The authors declare the following potential competing financial interests:

Natalia Novoradovskaya, Katherine Wilkins, Anne Lucas, Raymond Miller, and Carlos Pabon are all employees of Agilent Technologies, Inc. Agilent Sample B DNA reference sample is a current product and Sample A DNA is a potential product of Agilent Technologies, Inc.

Figures

**Fig. 1**
Overall flow diagram of process/method. Discovery of Class 1 variants came from consensus analysis of WES1/2/3/4 runs on overlapping WES kit target regions having high confidence. Additional Class 2 variants were discovered after analyzing WGS1 with WES results. Variants were confirmed by analyzing in silico A results where we combined individual BAMs from each cell line replicate and by analyzing merged-BAM Sample A from pooled Sample A individual replicate BAMs. Finally, a subset of these variants was orthogonally validated with ddPCR

**Fig. 2**
Defining the consensus target region (CTR). The regions shown are not to scale. Most of these regions and their sizes are provided in Additional file 1: Table S2. The low complexity regions are excluded from the CTR. Importantly, the size of the CTR is ~ 22.7 Mb for hg19

**Fig. 3**
ddPCR and WES concordance: a VAF concordance of individual cell line WES consensus results with ddPCR assays of that cell line. b Concordance (log₁₀ scale) of Sample A VAF between ddPCR and WES for positives only. c Various dilutions (C, D, E) of Sample A into B achieve the expected reduction in VAF as seen in the ddPCR results. It also shows the potential noise for measuring ddPCR variants below 0.1% (10^− 3) in the distribution of Sample B variants. d Concordance of replicate ddPCR assays (on log₁₀ scale) is very high (r² = .95) in diluted target Samples D and E. Putative VAF values from Sample B are also shown for comparison

**Fig. 4**
Illustration of considerations for determining positives and negatives within the reference material. Each WES kit coverage is shown relative to their intersection with coding regions (Interval4), the high confidence region, and the low complexity region. Also shown are known positive variant positions in Sample A (mostly Class 1 variants SNVs) including one identified by a violet box that is outside the Interval4 and CTR regions (Class 2 variant). Other positions shown include known negative positions

**Fig. 5**
a VAF histogram of Sample A variants (Class 1 and Class2) with the obvious large numbers of variants in the low VAF range from 0.01 to 0.10. b VAF histogram of normal Sample B which can be used to dilute variants from Sample A

See this image and copyright information in PMC

References

1. MAQC consortium. MicroArray/Sequencing Quality Control (MAQC/SEQC). U.S. Food and Drug Administration. 2019. https://www.fda.gov/science-research/bioinformatics-tools/microarraysequ.... Accessed 24 Feb 2020.
1. Zook JM, Catoe D, McDaniel J, Vang L, Spies N, Sidow A, Weng Z, Liu Y, Mason CE, Alexander N, Henaff E, McIntyre ABR, Chandramohan D, Chen F, Jaeger E, Moshrefi A, Pham K, Stedman W, Liang T, Saghbini M, Dzakula Z, Hastie A, Cao H, Deikus G, Schadt E, Sebra R, Bashir A, Truty RM, Chang CC, Gulbahce N, Zhao K, Ghosh S, Hyland F, Fu Y, Chaisson M, Xiao C, Trow J, Sherry ST, Zaranek AW, Ball M, Bobe J, Estep P, Church GM, Marks P, Kyriazopoulou-Panagiotopoulou S, Zheng GXY, Schnall-Levin M, Ordonez HS, Mudivarti PA, Giorda K, Sheng Y, Rypdal KB, Salit M. Extensive sequencing of seven human genomes to characterize benchmark reference materials. Sci Data. 2016;3(1):160025. doi: 10.1038/sdata.2016.25. - DOI - PMC - PubMed
1. Suzuki T, Tsukumo Y, Furihata C, Naito M, Kohara A. Preparation of the standard cell lines for reference mutations in cancer gene-panels by genome editing in HEK 293 T/17 cells. Genes and Environ. 2020;42:8. 10.1186/s41021-020-0147-2. - PMC - PubMed
1. Craig DW, Nasser S, Corbett R, Chan SK, Murray L, Legendre C, Tembe W, Adkins J, Kim N, Wong S, Baker A, Enriquez D, Pond S, Pleasance E, Mungall AJ, Moore RA, McDaniel T, Ma Y, Jones SJM, Marra MA, Carpten JD, Liang WS. A somatic reference standard for cancer genome sequencing. Sci Rep. 2016;6(1):24607. doi: 10.1038/srep24607. - DOI - PMC - PubMed
1. Kim J, Kim D, Lim JS, Maeng JH, Son H, Kang H-C, Nam H, Lee JH, Kim S. The use of technical replication for detection of low-level somatic mutations in next-generation sequencing. Nat Commun. 2019;10(1):1047. doi: 10.1038/s41467-019-09026-y. - DOI - PMC - PubMed

Publication types

Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

Substances

Actions

LinkOut - more resources

Full Text Sources
Other Literature Sources
- The Lens - Patent Citations Database
- scite Smart Citations
Medical
- MedlinePlus Health Information

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

A verified genomic reference sample for assessing performance of cancer panels detecting small variants of low allele frequency

Affiliations

A verified genomic reference sample for assessing performance of cancer panels detecting small variants of low allele frequency

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

Publication types

MeSH terms

Substances

LinkOut - more resources

Full Text Sources

Other Literature Sources

Medical