Scalable in-memory processing of omics workflows

Abstract

We present a proof of concept implementation of the in-memory computing paradigm that we use to facilitate the analysis of metagenomic sequencing reads. In doing so we compare the performance of POSIX™file systems and key-value storage for omics data, and we show the potential for integrating high-performance computing (HPC) and cloud native technologies. We show that in-memory key-value storage offers possibilities for improved handling of omics data through more flexible and faster data processing. We envision fully containerized workflows and their deployment in portable micro-pipelines with multiple instances working concurrently with the same distributed in-memory storage. To highlight the potential usage of this technology for event driven and real-time data processing, we use a biological case study focused on the growing threat of antimicrobial resistance (AMR). We develop a workflow encompassing bioinformatics and explainable machine learning (ML) to predict life expectancy of a population based on the microbiome of its sewage while providing a description of AMR contribution to the prediction. We propose that in future, performing such analyses in 'real-time' would allow us to assess the potential risk to the population based on changes in the AMR profile of the community.

Keywords: Bioinformatics; Cloud; HPC; Key-value store; Machine learning; Metagenomics.

Graphical abstract

Fig. 1

Architecture stack of the environment used to study in-memory workflows processing.

Fig. 2

Example bioinformatic workflow to enable analysis of metagenomic samples from untreated sewage.

Fig. 3

Workflow Implementation - Handling of a paired read sample in the proposed architecture. File I/O denotes reads and writes from/to a parallel file system. Socket I/O denotes reads and writes to an external in-memory key-value store.

Fig. 4

Performance profile of the base line pipeline for different input FASTQ file sizes.

Fig. 5

Schematic for in-memory duplicate removal using external key-value store.

Fig. 6

Performance profile of the external key-value store duplicate removal filter using DBR APIs (blue) and Hiredis APIs (orange). 1563892 FASTQ records, 220 bytes each have been used.

Fig. 7

Schematic for in-memory duplicate removal using MPI and a local memory approach.

Fig. 8

Performance profile of the local memory duplicate removal filter.

Fig. 9

Comparison of performance of ML regression analyses. Comparing ML model performance: (a) using the full dataset 223 samples x 7126 AMRs, (b) for the best model from (a) Random Forest with sequentially reduced numbers of features, (c) using the 223 samples x 30 selected AMRs from (b), (d) showing the true versus predicted values for the test dataset when processed through the Neural Network from (c).

Fig. 10

Local ML model explanation for 3 countries with the lowest life expectancy (a-c) and 3 countries with the highest life expectancy (d-f). Rows denote the AMR genes in ranked order from top to bottom according to the feature importance or impact on the models predictions. The values on the x-axis denote the SHAP calculated impact value of the related AMR gene on the models prediction of life expactancy in years for that sample.