IonCRAM: a reference-based compression tool for ion torrent sequence files

Abstract

Background: Ion Torrent is one of the major next generation sequencing (NGS) technologies and it is frequently used in medical research and diagnosis. The built-in software for the Ion Torrent sequencing machines delivers the sequencing results in the BAM format. In addition to the usual SAM/BAM fields, the Ion Torrent BAM file includes technology-specific flow signal data. The flow signals occupy a big portion of the BAM file (about 75% for the human genome). Compressing SAM/BAM into CRAM format significantly reduces the space needed to store the NGS results. However, the tools for generating the CRAM formats are not designed to handle the flow signals. This missing feature has motivated us to develop a new program to improve the compression of the Ion Torrent files for long term archiving.

Results: In this paper, we present IonCRAM, the first reference-based compression tool to compress Ion Torrent BAM files for long term archiving. For the BAM files, IonCRAM could achieve a space saving of about 43%. This space saving is superior to what achieved with the CRAM format by about 8-9%.

Conclusions: Reducing the space consumption of NGS data reduces the cost of storage and data transfer. Therefore, developing efficient compression software for clinical NGS data goes beyond the computational interest; as it ultimately contributes to the overall cost reduction of the clinical test. The space saving achieved by our tool is a practical step in this direction. The tool is open source and available at Code Ocean, github, and http://ioncram.saudigenomeproject.com .

Fig. 1

Flow Signals and their position in the SAM/BAM file. The upper part shows an example DNA fragment to be sequenced by an Ion Torrent machine. The key and bar code sequences are ligated (pre-pended) to the fragment. The key sequence (TCAG) is a control sequence to ensure correct sequencing. A barcode sequence is added to a certain group of fragments. The use of barcodes makes it possible to sequence the DNA of different samples/patients in one run. The lower part of the figure shows a schematic representation of the fields in the SAM file. The SAM file is the non-binary readable version of the BAM. The header part includes the flow cycle and the key sequence. Each line in the SAM file represents one read, aligned to the reference genome. The remaining rows include the read information in a tab-separated format: We show only the columns/fields of relevance to this paper. We show the fields including the read ID, the physical position and the CGAR string which represents the alignment, the bases of the DNA sequence in the read, the quality field, and the flow signals in the ZM field

Fig. 2

Base calling based on flow signals. The upper part shows an example DNA fragment to be sequenced. The second part shows the sequence of nucleotides in the flow cycle. It also shows the values of the sensed flow signals and the called bases. A flow signal value exceeding a certain threshold means that a base had hybridized to the template and the corresponding base in the flow cycle is reported. If the flow signal value doubles, this indicates a polymer of identical bases. The base calling software calibrates the signal values and decides the length of the polymer

Fig. 3

Compression Running times and space consumption. Compression running times and RAM consumption: Average running times for compressing (a) gene panels in seconds, (c) in-house exomes in minutes, (e) and public exomes in minutes. The measurements are for using Scramble and for using IonCRAM with the gzip, xz, and Zstd options. The average running time for gene panels is the average running times of the 11 gene panel files, and so did we for the set of the three public exomes and the set of four public exomes. The average RAM consumption in GB for gene panels, in-house exomes, and public exomes is shown in (b), (d), and (f)

Fig. 4

Decompression Running times and space consumption. Decompression running times and RAM consumption: Average running times for decompressing (a) gene panels in seconds, (c) in-house exomes in minutes, (e) and public exomes in minutes. The measurements are for using Scramble and for using IonCRAM with the gzip, xz, and Zstd options. The average running time for gene panels is the average running times of the 11 gene panel files, and so did we for the set of the three public exomes and the set of four public exomes. The average RAM consumption in GB for gene panels, in-house exomes, and public exomes is shown in (b), (d), and (f)