Cloud-based interactive analytics for terabytes of genomic variants data

Abstract

Motivation: Large scale genomic sequencing is now widely used to decipher questions in diverse realms such as biological function, human diseases, evolution, ecosystems, and agriculture. With the quantity and diversity these data harbor, a robust and scalable data handling and analysis solution is desired.

Results: We present interactive analytics using a cloud-based columnar database built on Dremel to perform information compression, comprehensive quality controls, and biological information retrieval in large volumes of genomic data. We demonstrate such Big Data computing paradigms can provide orders of magnitude faster turnaround for common genomic analyses, transforming long-running batch jobs submitted via a Linux shell into questions that can be asked from a web browser in seconds. Using this method, we assessed a study population of 475 deeply sequenced human genomes for genomic call rate, genotype and allele frequency distribution, variant density across the genome, and pharmacogenomic information.

Availability and implementation: Our analysis framework is implemented in Google Cloud Platform and BigQuery. Codes are available at https://github.com/StanfordBioinformatics/mvp_aaa_codelabs.

Contact: cuiping@stanford.edu or ptsao@stanford.edu.

Supplementary information: Supplementary data are available at Bioinformatics online.

Fig. 1.

The computational paradigm of cloud platform-based data processing and Dremel-based interactive analytics for large-scale genomic data. Shown here is the Google Cloud Platform-enabled solution. Variant calling from raw reads to genotypes is performed by GATK via Google Genomics API in Compute Engine Virtual Machines (VMs). Genomic data is represented in a Dremel database to enable interactive data QC and analytics

Fig. 2.

Callability assessment. (A) Diagram shows the overall categories of genomic positions by callability and quality. (B) Percentage of detected genomic positions in each chromosome for each genome in this WGS study, based on all calls reported by GATK. (C) Uncalled regions (URs) and the length distribution, (left) number of URs per chromosome in each genome, categorized by different length groups; (middle) number of URs per chromosome in each genome, normalized by chromosome lengths; (right): commonality of URs across all genomes. (D) Numbers of SNVs and INDELs passing different QC levels. No_QC: all calls by GATK without any filtering. Seq_QC: calls passing the VQSR filtering in GATK. Full_QC: calls passing all levels of QC. (E) Percentage of genomic positions called as reference bases, SNVs and INDELs

Fig. 3.

Saturation call rate for SNVs and INDELs. (A) Number of unique SNVs and INDELs by increasing number of genomes. (B) Distribution of allele frequencies for SNVs. (C) Distribution of allele frequencies for INDELs

Fig. 4.

Dremel query performance assessment on real and simulated genomic data. Shown here are the query run times and cost for five representative queries on tables containing real genomic data, ranging from 5 to 461 genomes (A and C), and simulated genomic data, ranging from 1000 to 5000 genomes (B and D)