Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 May 1;9(5):giaa042.
doi: 10.1093/gigascience/giaa042.

MaRe: Processing Big Data with application containers on Apache Spark

Marco Capuccini  1   2 , Martin Dahlö  2   3   4 , Salman Toor  1 , Ola Spjuth  2
Affiliations

MaRe: Processing Big Data with application containers on Apache Spark

Marco Capuccini et al. Gigascience. .

Abstract

Background: Life science is increasingly driven by Big Data analytics, and the MapReduce programming model has been proven successful for data-intensive analyses. However, current MapReduce frameworks offer poor support for reusing existing processing tools in bioinformatics pipelines. Furthermore, these frameworks do not have native support for application containers, which are becoming popular in scientific data processing.

Results: Here we present MaRe, an open source programming library that introduces support for Docker containers in Apache Spark. Apache Spark and Docker are the MapReduce framework and container engine that have collected the largest open source community; thus, MaRe provides interoperability with the cutting-edge software ecosystem. We demonstrate MaRe on 2 data-intensive applications in life science, showing ease of use and scalability.

Conclusions: MaRe enables scalable data-intensive processing in life science with Apache Spark and application containers. When compared with current best practices, which involve the use of workflow systems, MaRe has the advantage of providing data locality, ingestion from heterogeneous storage systems, and interactive processing. MaRe is generally applicable and available as open source software.

Keywords: Apache Spark; Big Data; MapReduce; application containers; workflows.

PubMed Disclaimer

Figures

Figure 1:
Figure 1:
Execution diagram for the map primitive. The primitive takes an RDD that is partitioned over N nodes, it transforms each partition using a Docker container, and it returns a new RDD′. The logic is implemented using mapPartitions from the RDD API. Because mapPartitions generates a single stage, data are not shuffled between nodes.
Figure 2:
Figure 2:
Execution diagram for the reduce primitive. The primitive takes an input RDD, partitioned over N nodes, and it iteratively aggregates records using a Docker container, reducing the number of partitions until an RDD′, containing a single result partition, is returned. The logic is implemented using mapPartitions and repartition from the RDD API, to aggregate records in partitions and to decrease the number of partitions, respectively. Because repartition is called in each of the K iterations, K stages are generated, giving place to K data shuffles.
Figure 3:
Figure 3:
WSE for the VS application implemented in MaRe (Listing 2). The results are produced by using SureChEMBL as input, and we show the WSE when using tmpfs and ext4 as temporary mount point for passing the data to the containers.
Figure 4:
Figure 4:
SSE for the SNP calling implemented in MaRe (Listing 3). The results are produced by using a full individual dataset from the 1KGP as input.
Figure 5:
Figure 5:
SSE for the SNP calling alignment stage implemented in MaRe (Listing 3, lines 1–13). The results are produced by using a full individual dataset from the 1KGP as input, and we show the SSE when using an SSD-based, ext4 temporary mount point as well as Unix pipes for passing the data to the containers.
See this image and copyright information in PMC

References

    1. Stephens ZD, Lee SY, Faghri F, et al.. Big data: astronomical or genomical?. PLoS Biol. 2015; 13(7):e1002195. - PMC - PubMed
    1. Foster LJ, DeMarco ML. At the intersection of proteomics and big data science. Clin Chem. 2017;63(10):1663. - PubMed
    1. Peters K, Bradbury J, Bergmann S, et al.. PhenoMeNal: Processing and analysis of metabolomics data in the cloud. Gigascience. 2018;8(2):giy149. - PMC - PubMed
    1. Peng H. Bioimage informatics: a new area of engineering biology. Bioinformatics. 2008;24(17):1827–36. - PMC - PubMed
    1. Brown N, Cambruzzi J, Cox PJ, et al.. Big Data in drug discovery. Prog Med Chem. 2018;57:277–356. - PubMed

Publication types