An automated real-time integration and interoperability framework for bioinformatics

Abstract

Background: In recent years data integration has become an everyday undertaking for life sciences researchers. Aggregating and processing data from disparate sources, whether through specific developed software or via manual processes, is a common task for scientists. However, the scope and usability of the majority of current integration tools fail to deal with the fast growing and highly dynamic nature of biomedical data.

Results: In this work we introduce a reactive and event-driven framework that simplifies real-time data integration and interoperability. This platform facilitates otherwise difficult tasks, such as connecting heterogeneous services, indexing, linking and transferring data from distinct resources, or subscribing to notifications regarding the timeliness of dynamic data. For developers, the framework automates the deployment of integrative and interoperable bioinformatics applications, using atomic data storage for content change detection, and enabling agent-based intelligent extract, transform and load tasks.

Conclusions: This work bridges the gap between the growing number of services, accessing specific data sources or algorithms, and the growing number of users, performing simple integration tasks on a recurring basis, through a streamlined workspace available to researchers and developers alike.

Fig. 1

Framework architecture highlighting the different system layers. a external Original Resources are accessed for data extraction; b local or distributed Agents poll Original Resources; c the internal Data store uses a relational database (PostgreSQL or MySQL) to store data and an object cache (Redis) for improved performance; d the application engine, is implemented in Ruby, with the Rails framework, and controls the entire application and its API; e the Postman applies the data extracted by the Agents to the Templates and executes the final delivery; f the external Destination Resources receive the data from the system.

Fig. 2

Framework monitoring and integration sequence diagram. In addition to the listed steps, all actions are logged internally for auditing, error tracking and performance analysis. Two alternative pipelines can be executed: a distributed agents generate a different sequence from step 3, where FluxCapacitor mediates all interactions; b events data can be pushed directly into the platform, generating a new sequence starting at step 7.

Fig. 3

Applying data transformations. Data from Original Resources (in CSV/TSV, XML, JSON or SQL) can be easily translated and transformed (into URL requests, files, SQL queries or emails) using the framework’s templates: a CSV data are automatically inserted into a SQL database; b data are extracted from a SQL query into a CSV file; c XML elements are extracted (using XPath) and sent to a web service via POST request.

Fig. 4

Web interface for proposed platform prototype. This interface highlights the integration configuration for automating human variome integration. This integration features one agent (LOVD XML Agent) and one template (SQL variant). The former configures how to extract mutation data from LOVD API and the latter specifies the configuration for storing extracted data in a relational database.