Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2014 Sep 29:2014:bau093.
doi: 10.1093/database/bau093. Print 2014.

The Cancer Genomics Hub (CGHub): overcoming cancer through the power of torrential data

Christopher Wilks  1 , Melissa S Cline  2 , Erich Weiler  2 , Mark Diehkans  2 , Brian Craft  2 , Christy Martin  2 , Daniel Murphy  2 , Howdy Pierce  2 , John Black  2 , Donavan Nelson  2 , Brian Litzinger  2 , Thomas Hatton  2 , Lori Maltbie  2 , Michael Ainsworth  2 , Patrick Allen  2 , Linda Rosewood  2 , Elizabeth Mitchell  2 , Bradley Smith  2 , Jim Warner  2 , John Groboske  2 , Haifang Telc  2 , Daniel Wilson  2 , Brian Sanford  2 , Hannes Schmidt  2 , David Haussler  2 , Daniel Maltbie  2
Affiliations

The Cancer Genomics Hub (CGHub): overcoming cancer through the power of torrential data

Christopher Wilks et al. Database (Oxford). .

Abstract

The Cancer Genomics Hub (CGHub) is the online repository of the sequencing programs of the National Cancer Institute (NCI), including The Cancer Genomics Atlas (TCGA), the Cancer Cell Line Encyclopedia (CCLE) and the Therapeutically Applicable Research to Generate Effective Treatments (TARGET) projects, with data from 25 different types of cancer. The CGHub currently contains >1.4 PB of data, has grown at an average rate of 50 TB a month and serves >100 TB per week. The architecture of CGHub is designed to support bulk searching and downloading through a Web-accessible application programming interface, enforce patient genome confidentiality in data storage and transmission and optimize for efficiency in access and transfer. In this article, we describe the design of these three components, present performance results for our transfer protocol, GeneTorrent, and finally report on the growth of the system in terms of data stored and transferred, including estimated limits on the current architecture. Our experienced-based estimates suggest that centralizing storage and computational resources is more efficient than wide distribution across many satellite labs. Database URL: https://cghub.ucsc.edu.

PubMed Disclaimer

Figures

Figure 1.
Figure 1.
CGHub and GeneTorrent conceptual system design and flow. (1) Client retrieves a list of downloadable files from the CGHub Web services. (2) Client uses GT to initiate a download. (3) Download is handed to GT distributor after proper security checks have passed. (4) Download is distributed to multiple transfer servers within a pool of available servers. (5) Servers announce themselves to the tracker as serving the requested file(s). (6) Client gets list of servers from the tracker. (7) Client downloads data directly from assigned transfer servers. All sequence data are read from the distributed GPFS.
Figure 2.
Figure 2.
CGHub client security transaction: (1) CGHub retrieves the authorized list of users from the NCI DAC multiple times a day. (2) The client accesses CGHub’s Web page to login and is redirected to NIH’s InCommon federated SSO authentication page over HTTPS where the client logs in using previously issued NIH credentials. (3) A successful authentication is sent back to the client via Shibboleth. (4) The client then sends the authentication via Shibboleth to CGHub’s Web server. (5) CGHub’s Web server then generates a unique cryptographically encoded key for the client, which has a 1-year expiration from the date of issuance, and sends it securely to the client over HTTPS. (6) The client then sends his\her file request with its encrypted identity key from Step 5 over HTTPS to CGHub’s security service. (7) If the client’s key is recognized and authorized to download the requested file, CGHub’s security service will send a signed temporary x509 certificate specific to the requested file back to the client over HTTPS. (8) The client then sends the x509 certificate from Step 7 to the CGHub data server to prove the client’s identity and authorization to download the requested file. (9) The CGHub data server verifies the certificate and establishes an SSL connection using AES-256 for encryption for the file transfer to the client.
Figure 3.
Figure 3.
CGHub’s multi-layered parallelized transfer architecture using GT.
Figure 4.
Figure 4.
Comparison of transfer performance between sFTP, GridFTP and GT with varying numbers of TCP connections and GT server instances.
Figure 5.
Figure 5.
CGHub growth measured by size (GB) of uploads, downloads and number of users per month. The y-axis is in log scale.
Figure 6.
Figure 6.
CGHub cumulative growth: downloaded vs. stored data.
Figure 7.
Figure 7.
CGHub’s outbound firewall usage 1/2012–5/2014 (averaged 24-h intervals, some precision is lost).
See this image and copyright information in PMC

References

    1. Kent W.J., Haussler D. (2001) Assembly of the working draft of the human genome with GigAssembler. Genome Res., 11, 1541–1548 - PMC - PubMed
    1. Wilks C., Maltbie D., Diekhans M., et al. (2013) CGHub: Kick-starting the Worldwide Genome Web. Proc. Asia Pac. Adv. Netw., 35, 1–13
    1. Moore G. (1998). Cramming more components onto integrated circuits. Proc IEEE, 86, 82–85
    1. Li H., Handsaker B., Wysoker A., et al. (2009) The Sequence alignment/map (SAM) format and SAMtools. Bioinformatics, 25, 2078–2079 - PMC - PubMed
    1. National Institute of Standards and Technology. (2012) Secure Hash Standard, FIPS180-4. NIST, Gaithersburg, MD

Publication types