BioConvert: a comprehensive format converter for life sciences

Abstract

Bioinformatics is a field known for the numerous standards and formats that have been developed over the years. This plethora of formats, sometimes complementary, and often redundant, poses many challenges to bioinformatics data analysts. They constantly need to find the best tool to convert their data into the suitable format, which is often a complex, technical and time consuming task. Moreover, these small yet important tasks are often difficult to make reproducible. To overcome these difficulties, we initiated BioConvert, a collaborative project to facilitate the conversion of life science data from one format to another. BioConvert aggregates existing software within a single framework and complemented them with original code when needed. It provides a common interface to make the user experience more streamlined instead of having to learn tens of them. Currently, BioConvert supports about 50 formats and 100 direct conversions in areas such as alignment, sequencing, phylogeny, and variant calling. In addition to being useful for end-users, BioConvert can also be utilized by developers as a universal benchmarking framework for evaluating and comparing numerous conversion tools. Additionally, we provide a web server implementing an online user-friendly interface to BioConvert, hence allowing direct use for the community.

Figure 1.

Template of a new converter performing conversion from format A to format B. Methods are implemented using Python and optionally external binaries.

Figure 2.

Example of an implicit conversion where extensions suffice for inferring the type of conversion required.

Figure 3.

Example of an explicit conversion where extensions can not be resolved automatically.

Figure 4.

Example of an explicit conversion with an implicit output.

Figure 5.

Example of an explicit conversion with an explicit output.

Figure 6.

Formats and conversions available in BioConvert are represented as a directed acyclic graph. Nodes correspond to formats and edges correspond to conversions. Colors indicate the degree of each format (number of connections/conversions that a node/format has in the graph).

Figure 7.

The formats included in BioConvert cover NGS formats. In this graph, nodes (formats) are clustered according to their field of expertise. We could identify several topics including variant calling, phylogeny, sequencing data, alignments, ...

Figure 8.

Example of conversion that produces two output files.

Figure 9.

Number of methods implemented in each conversion. Most have only 1 or 2 methods.

Figure 10.

Single-mode benchmarking. BioConvert provides a sub-command to compare the computational time of all methods available within a given converter (here FastQ to FastA). Each method is run several times to estimate the average time for each method as well as the standard errors. In this instance, the mawk method gives the best performance. Results and error bars may fluctuate depending on hardware performances and concurrent running processes. Benchmark obtained with a SSD hard disk, with compressed input and uncompressed output files.

Figure 11.

The multi-mode benchmark of the fastq2fasta conversion involves repeating the single-mode benchmark multiple times to better understand the variability within a method and provide more confidence in determining the fastest method. The mawk method has consistently low variation and is one of the fastest methods.

Figure 12.

Benchmarking of the bam2sam converter with four methods implemented in BioConvert. The only difference between top and bottom panels is related to the version of samtools used for benchmark (1.7 and 1.15 respectively). In the top panel, the performance of samtools and sambamba were similar, while in the bottom panel, samtools was 2-3 times faster. This difference can be attributed to the updated version of samtools resulting in a significantly increased performance.