The GA4GH Task Execution Application Programming Interface: Enabling Easy Multicloud Task Execution

Abstract

The Global Alliance for Genomics and Health (GA4GH) Task Execution Service (TES) application programming interface (API) is a standardized schema and API for describing and executing batch execution tasks. It provides a common way to submit and manage tasks to a variety of compute environments, including on-premises high-performance computing and high-throughput computing systems, cloud computing platforms, and hybrid environments. The TES API is designed to be flexible and extensible, allowing it to be adapted to a wide range of use cases, such as "bringing compute to the data" solutions for federated and distributed data analysis, or load balancing across multicloud infrastructures. This API has been adopted by numerous different service providers and is utilized by several workflow engines, yielding a single abstracted interface for developers and researchers. Using its capabilities, genome research institutes are building extensible hybrid compute systems to study life science.

FIGURE 1.

Common TES use cases. The TES API wraps around compute environments providing a standard way of executing tasks. Researchers can write and package their tasks and data in a domain-specific language (DSL) workflow language. They then hand the orchestration of the tasks over to the respective workflow management systems. The workflow management systems can then make use of TES clients to distribute tasks across different environments. Alternatively, users can submit individual tasks to TES servers directly via command-line interfaces (CLIs) or GUI interfaces. Thus, TES makes it easier for researchers to make use of a variety of compute environments seamlessly. Applications can support new compute environments by integrating with TES APIs, rather than develop unique connections for each environment. CWL: Common Workflow Language.

FIGURE 2.

An example TES packet. The example TES task demonstrates the use of inputs, outputs, and logging.

FIGURE 3.

The TES execution architecture. The figure presents an outline of the separate layers found in current TES service implementations. The client talks to a server, which is responsible for allocating a worker node on an HPC/HTC or cloud infrastructure. The TES worker is responsible for transferring inputs, running user code, capturing logging, and storing outputs.