Hypermedia-based software architecture enables Test-Driven Development

Abstract

Objectives: Using agile software development practices, develop and evaluate an architecture and implementation for reliable and user-friendly self-service management of bioinformatic data stored in the cloud.

Materials and methods: Comprehensive Oncology Research Environment (CORE) Browser is a new open-source web application for cancer researchers to manage sequencing data organized in a flexible format in Amazon Simple Storage Service (S3) buckets. It has a microservices- and hypermedia-based architecture, which we integrated with Test-Driven Development (TDD), the iterative writing of computable specifications for how software should work prior to development. Relying on repeating patterns found in hypermedia-based architectures, we hypothesized that hypermedia would permit developing test "templates" that can be parameterized and executed for each microservice, maximizing code coverage while minimizing effort.

Results: After one-and-a-half years of development, the CORE Browser backend had 121 test templates and 875 custom tests that were parameterized and executed 3031 times, providing 78% code coverage.

Discussion: Architecting to permit test reuse through a hypermedia approach was a key success factor for our testing efforts. CORE Browser's application of hypermedia and TDD illustrates one way to integrate software engineering methods into data-intensive networked applications. Separating bioinformatic data management from analysis distinguishes this platform from others in bioinformatics and may provide stable data management while permitting analysis methods to advance more rapidly.

Conclusion: Software engineering practices are underutilized in informatics. Similar informatics projects will more likely succeed through application of good architecture and automated testing. Our approach is broadly applicable to data management tools involving cloud data storage.

Keywords: cloud computing; data management; high-throughput nucleotide sequencing; software.

Figure 1.

Illustration of Test-Driven Development, sometimes called red-green-refactor.

Figure 2.

CORE Browser architecture stack diagram. CORE Browser implements a microservices architecture. Microservices are color-coded according to their data store: blue for MongoDB; orange for Keycloak, which stores user data in a MySQL database; and gray for AWS S3. They are deployed with Docker and exchange messages asynchronously using a RabbitMQ message queue, color-coded black to indicate that all microservices have them as dependencies. The web client code uses an internally developed Angular widget and style library, hci-ng, and the third-party ag-grid and Fontawesome libraries.

Figure 3.

REST API convention in CORE Browser, illustrated with the Organizations microservice. The table, upper-left, shows the routes for getting (GET), updating (PUT), creating (POST), and deleting (DELETE) organizations. GET calls respond with JSON-formatted data, including an organization object, metadata for presenting it on screen, and links to actions that can be performed on the organization, upper-right. Links have a prompt property, one or more relation types (rel) that indicate the nature of the link, and the href of the link. Link hrefs are truncated in this figure to save space. The “opener” route responds with a link for getting the organization’s AWS accounts, lower-left.

Figure 4.

The network of links in CORE Browser. The orange, blue, and gray boxes represent the objects and data displayed in the application. The boxes are color-coded by where they are stored in the backend. The arrows represent links.

Figure 5.

Screenshot of CORE Browser’s object explorer page, with annotations to illustrate how responses to the links in Figure 4 are rendered on screen. The user has logged into the application (#1), and they are authorized to access data from certain labs (organizations, #2). Upon selecting an organization in the dropdown, they are shown a tree of accounts (#3), from which they can open buckets (#4), folders (#5), and files (#6). Depending on the relation types in an object’s links, the web client might treat the object as a folder to be opened (accounts, buckets, and folders), or it might render the links as actions to be performed (#7, links to open, duplicate, move, or delete folders and files).