The ENCODE Uniform Analysis Pipelines

Abstract

The Encyclopedia of DNA elements (ENCODE) project is a collaborative effort to create a comprehensive catalog of functional elements in the human genome. The current database comprises more than 19000 functional genomics experiments across more than 1000 cell lines and tissues using a wide array of experimental techniques to study the chromatin structure, regulatory and transcriptional landscape of the Homo sapiens and Mus musculus genomes. All experimental data, metadata, and associated computational analyses created by the ENCODE consortium are submitted to the Data Coordination Center (DCC) for validation, tracking, storage, and distribution to community resources and the scientific community. The ENCODE project has engineered and distributed uniform processing pipelines in order to promote data provenance and reproducibility as well as allow interoperability between genomic resources and other consortia. All data files, reference genome versions, software versions, and parameters used by the pipelines are captured and available via the ENCODE Portal. The pipeline code, developed using Docker and Workflow Description Language (WDL; https://openwdl.org/) is publicly available in GitHub, with images available on Dockerhub (https://hub.docker.com), enabling access to a diverse range of biomedical researchers. ENCODE pipelines maintained and used by the DCC can be installed to run on personal computers, local HPC clusters, or in cloud computing environments via Cromwell. Access to the pipelines and data via the cloud allows small labs the ability to use the data or software without access to institutional compute clusters. Standardization of the computational methodologies for analysis and quality control leads to comparable results from different ENCODE collections - a prerequisite for successful integrative analyses.

Keywords: NGS analysis; analysis pipelines; software.

Figure 1.

Pipeline infrastructure and continuous integration.

Figure 2

A) Demo WDL pipeline and B) CROO JSON that defines how to organize and display outputs

Figure 3.

Croo HTML report example showing file table, task graph, and link to UCSC genome browser. The red boxes represent raw data files, the blue boxes represent software steps (abstract names), and the yellow boxes represent intermediate or output processed data files.

Figure 4

Pipelines for ChIP-seq and ATAC-seq A) TF ChIP-seq schematic; (https://www.encodeproject.org/pipelines/ENCPL367MAS/, B) Histone ChIP-seq schematic; (https://www.encodeproject.org/pipelines/ENCPL612HIG/), ATAC-seq schematic; https://www.encodeproject.org/pipelines/ENCPL787FUN/). Not shown: schematic pipelines for unreplicated experiments; TF ChIP-seq; https://www.encodeproject.org/pipelines/ENCP481MLO/, Histone ChIP-seq; https://www.encodeproject.org/pipelines/ENCPL809GEM/. ATAC-seq : https://www.encodeproject.org/pipelines/ENCPL344QWT/

Figure 5

Pipeline for RNA-seq A), bulk RNA seq schematic (https://www.encodeproject.org/pipelines/ENCPL862USL/) B) micro-RNA-seq schematic (https://www.encodeproject.org/pipelines/ENCPL280YDY/) C) long-read RNA-seq schematic (https://www.encodeproject.org/pipelines/ENCPL239OZU/) D) RAMPAGE (and CAGE) schematic (https://www.encodeproject.org/pipelines/ENCPL122WIM)

Figure 6

Pipeline schematic using gemBS for whole-genome bisulfite sequencing (https://www.encodeproject.org/pipelines/ENCPL182IUX/)

Figure 7

Pipeline schematic for DNase-seq (https://www.encodeproject.org/pipelines/ENCPL848KLD)

Figure 8:

Pipeline schematic for Hi-C pipeline A) Juicer mapping and contact maps schematic: (https://encodeproject.org/pipelines/ENCPL839OAB/). Megamapping is the same but starting from arrays of .hic and .bigWig files merged into deeper maps. B) Genophasing schematic (https://www.encodeproject.org/pipelines/ENCPL780XND/) C) Diploidification schematic (https://www.encodeproject.org/pipelines/ENCPl478DPO/)