fastp: an ultra-fast all-in-one FASTQ preprocessor

Abstract

Motivation: Quality control and preprocessing of FASTQ files are essential to providing clean data for downstream analysis. Traditionally, a different tool is used for each operation, such as quality control, adapter trimming and quality filtering. These tools are often insufficiently fast as most are developed using high-level programming languages (e.g. Python and Java) and provide limited multi-threading support. Reading and loading data multiple times also renders preprocessing slow and I/O inefficient.

Results: We developed fastp as an ultra-fast FASTQ preprocessor with useful quality control and data-filtering features. It can perform quality control, adapter trimming, quality filtering, per-read quality pruning and many other operations with a single scan of the FASTQ data. This tool is developed in C++ and has multi-threading support. Based on our evaluation, fastp is 2-5 times faster than other FASTQ preprocessing tools such as Trimmomatic or Cutadapt despite performing far more operations than similar tools.

Availability and implementation: The open-source code and corresponding instructions are available at https://github.com/OpenGene/fastp.

Fig. 1.

Workflow of fastp. (a) Main workflow of paired-end data processing, and (b) paired-end preprocessor of one read pair. In the main workflow, a pair of FASTQ files is loaded and packed, after which each read pair is processed individually in the paired-end preprocessor, described in (b)

Fig. 2.

A demonstration of extending an adapter seed in both forward and backward directions. The found adapter is GCAAATCGATCGACT, with the first two bases (GC) as the upstream sequence, the central ten bases as the adapter seed, and the last three bases (ACT) as the downstream sequence

Fig. 3.

The base content ratio curves generated by fastp for one Illumina NextSeq FASTQ file. (a) Before fastp preprocessing, and (b) after fastp preprocessing. As depicted in (a), the G curve is abnormal and the G/C curves are separated. In (b), the G/C separation problem is eliminated

Fig. 4.

Duplication estimation. The read percentages and mean GC ratios of different duplication levels. The mean GC ratio curve is truncated since the reads with higher duplication level are too few to compute a stable mean value

Fig. 5.

Overrepresented sequence analysis results. The right column shows the histogram of occurrence among all sequencing cycles

Fig. 6.

Result of adapter trimming performance evaluation. The X-axis is the number of allowed mismatches when searching for suspected adapter sequences, and the Y-axis is the count of suspected adapter sequences