Standardized and accessible multi-omics bioinformatics workflows through the NMDC EDGE resource

Abstract

Accessible and easy-to-use standardized bioinformatics workflows are necessary to advance microbiome research from observational studies to large-scale, data-driven approaches. Standardized multi-omics data enables comparative studies, data reuse, and applications of machine learning to model biological processes. To advance broad accessibility of standardized multi-omics bioinformatics workflows, the National Microbiome Data Collaborative (NMDC) has developed the Empowering the Development of Genomics Expertise (NMDC EDGE) resource, a user-friendly, open-source web application (https://nmdc-edge.org). Here, we describe the design and main functionality of the NMDC EDGE resource for processing metagenome, metatranscriptome, natural organic matter, and metaproteome data. The architecture relies on three main layers (web application, orchestration, and execution) to ensure flexibility and expansion to future workflows. The orchestration and execution layers leverage best practices in software containers and accommodate high-performance computing and cloud computing services. Further, we have adopted a robust user research process to collect feedback for continuous improvement of the resource. NMDC EDGE provides an accessible interface for researchers to process multi-omics microbiome data using production-quality workflows to facilitate improved data standardization and interoperability.

Keywords: Bioinformatics workflows; Microbiome; Multi-omics; Open-source; Software; Standardization.

Fig. 1

The NMDC EDGE architecture has three layers: the web application (A, red), orchestration (B, purple), and execution (C, blue) layers. The web application layer and orchestration layer run in a virtual machine (VM) and the execution layer runs in a high-performance computing (HPC) environment. (A) Users engage with the web application to select input files and workflows. This information is used to populate a workflow template WDL file and an input json file. (B) The orchestration layer generates a Cromwell job for the execution layer. Cromwell jobs are executed in a shared Linux computing environment together with non-Cromwell jobs, where the resource manager (e.g., Slurm) handles the actual computing jobs and resources in the cluster. (C) Cromwell handles the packaging of WDL-defined workflows into Slurm jobs and also monitors the job execution and updates the MySQL database with the status. When jobs are completed, Slurm resumes control, cleans up temporary files, and writes standard outputs and error messages. Workflow output files are moved to destinations as defined by the WDL files on a filesystem shared between the web application and the execution layer. These then become accessible to the project owner and any users who have been granted access by the owner. The web application layer tracks the status of the workflow (e.g., queuing time, job status) through communications with the orchestration layer and updates the status for the user. In the event of a workflow failure, an error message is displayed to the user, giving information about the source of the failure.

Fig. 2

The NMDC standardized bioinformatics workflows and their associated inputs and main outputs for (A) sequencing data and (B) mass spectrometry data. Additional output files and visualizations are shown in Table 1. The rectangles indicate workflows, and gray parallelograms indicate inputs and outputs. Direct infusion Fourier-transform ion cyclotron mass spectrometry (DI FT-ICR MS); Metagenome-assembled genomes (MAGs); Liquid chromatography-tandem mass spectrometry (LC-MS/MS).

Fig. 3

Key features of the NMDC EDGE resource for running standardized NMDC bioinformatics workflows.