Cloud computing for genomic data analysis and collaboration

Abstract

Next-generation sequencing has made major strides in the past decade. Studies based on large sequencing data sets are growing in number, and public archives for raw sequencing data have been doubling in size every 18 months. Leveraging these data requires researchers to use large-scale computational resources. Cloud computing, a model whereby users rent computers and storage from large data centres, is a solution that is gaining traction in genomics research. Here, we describe how cloud computing is used in genomics for research and large-scale collaborations, and argue that its elasticity, reproducibility and privacy features make it ideally suited for the large-scale reanalysis of publicly available archived data, including privacy-protected data.

Figure 1:

Four doublings of the Sequence Read Archive from July 2012 to March 2017. The large jump in October 2016 is chiefly due to the TopMed project. As of June 2017, the SRA contains over 12 petabases (millions of billions of bases) of data.

Figure 2:

Elasticity allows the user to rent resources while paying only for what gets used. Panel (a) illustrates a scenario with two computational tasks to perform, colored red and blue. The red task requires 36 computer-hours and runs on up to 8 computers simultaneously. The blue task requires 18 computer-hours and runs on 3 computers simultaneously. On a smaller cluster (left) both the tasks run sequentially and require 15 hours to complete. On a larger cluster (right), representing a cloud cluster, the tasks can run simultaneously and the red task can use its full complement of 8 computers. As a result, both complete within 6 hours. This ignores the fact that many more users are contending for cloud clusters than are contending for an institutional cluster. The greater number of users is mitigated by the fact that needs and timing vary from user to user. Cloud providers also provide incentives, such as spot pricing, to encourage renting at less busy times.

Figure 3:

Each site (blue rectangle) has some computational resources and also generates a portion of the data (red puzzle pieces). In (a) analysis that require the full datasets are to be performed at multiple sites, requiring each of these sites to gather all portions of the data. As more sites join the analysis, more copies must be made. (b) and (c) are alternate solutions. In (b) sites consolidate their data in a cloud-based data center, where all analyses are performed. In (c), multiple sites organize themselves into a federated cloud, where each analysis of the full dataset is automatically coordinated to minimize data transfer. Where possible, the computers located where data are generated are used to analyze that subset.