DDBJ Database updates and computational infrastructure enhancement

Abstract

The Bioinformation and DDBJ Center (https://www.ddbj.nig.ac.jp) in the National Institute of Genetics (NIG) maintains a primary nucleotide sequence database as a member of the International Nucleotide Sequence Database Collaboration (INSDC) in partnership with the US National Center for Biotechnology Information and the European Bioinformatics Institute. The NIG operates the NIG supercomputer as a computational basis for the construction of DDBJ databases and as a large-scale computational resource for Japanese biologists and medical researchers. In order to accommodate the rapidly growing amount of deoxyribonucleic acid (DNA) nucleotide sequence data, NIG replaced its supercomputer system, which is designed for big data analysis of genome data, in early 2019. The new system is equipped with 30 PB of DNA data archiving storage; large-scale parallel distributed file systems (13.8 PB in total) and 1.1 PFLOPS computation nodes and graphics processing units (GPUs). Moreover, as a starting point of developing multi-cloud infrastructure of bioinformatics, we have also installed an automatic file transfer system that allows users to prevent data lock-in and to achieve cost/performance balance by exploiting the most suitable environment from among the supercomputer and public clouds for different workloads.

Figure 1.

General architecture of the NIG supercomputer installed in 2019. Based on the previous system, the NIG Supercomputer 2019 mainly consists of a distributed memory HPC cluster, high-performance parallel distributed file systems for calculation, and large capacity archiving storage systems for the DNA database. Those systems are interconnected via a high-throughput low-latency network (InfiniBand) and various management networks (Ethernet).

Figure 2.

Automatic file transfer system between the NIG supercomputer and a public cloud (Amazon Web Service). Dedicated data transfer server (Fusic data transfer) is installed in the NIG supercomputer that allows users to send data, up and down compute instances, running jobs and make configuration changes on the AWS cloud by using a series of command line tools installed on the NIG supercomputer. SINET5 network is subject to discount for egress network traffic charge of the public cloud.