BaseNet: A transformer-based toolkit for nanopore sequencing signal decoding

Abstract

Nanopore sequencing provides a rapid, convenient and high-throughput solution for nucleic acid sequencing. Accurate basecalling in nanopore sequencing is crucial for downstream analysis. Traditional approaches such as Hidden Markov Models (HMM), Recurrent Neural Networks (RNN), and Convolutional Neural Networks (CNN) have improved basecalling accuracy but there is a continuous need for higher accuracy and reliability. In this study, we introduce BaseNet (https://github.com/liqingwen98/BaseNet), an open-source toolkit that utilizes transformer models for advanced signal decoding in nanopore sequencing. BaseNet incorporates both autoregressive and non-autoregressive transformer-based decoding mechanisms, offering state-of-the-art algorithms freely accessible for future improvement. Our research indicates that cross-attention weights effectively map the relationship between current signals and base sequences, joint loss training through adding a pair of forward and reverse decoder facilitate model converge, and large-scale pre-trained models achieve superior decoding accuracy. This study helps to advance the field of nanopore sequencing signal decoding, contributes to technological advancements, and provides novel concepts and tools for researchers and practitioners.

Keywords: Basecall; Machine learning algorithm; Nanopore sequencing; Transformer.

Graphical abstract

Fig. 1

Summary of benchmark dataset characteristics. (a) Data size statistics of each species in the dataset. (b) Read length distribution statistics of each species in the dataset. (c) Statistics of different bases proportion in the dataset.

Fig. 2

Fast attention mechanism in BaseNet. This schematic illustrates the fast attention mechanism used in BaseNet. The method involves training separate matrices, α and β, each with dimensions [N, 1], for Q and K, respectively. These matrices are used to perform a weighted summations on Q and K, resulting in transformed matrices Q' and K'. The transformed matrices are then multiplied together, reducing the computational complexity to O(N·d).

Fig. 3

Autoregressive Transformer-based model architecture for nanopore sequencing. The schematic illustrates the architecture of the autoregressive transformer model tailored for nanopore sequencing data. The model includes convolutional modules for feature extraction and down-sampling, an encoder composed of 8 layers for context modeling, and a decoder with 8 layers for sequence generation.

Fig. 4

Self-supervised large-scale model architecture in BaseNet. The schematic illustrates the self-supervised large-scale model architecture developed in BaseNet. The model comprises three key components: a feature extraction module, an encoder module for context modeling, and a quantization module for learning discrete common features through self-supervised pre-training.

Fig. 5

Rescore and joint loss training model in BaseNet. This schematic illustrates the rescore and joint loss training model used in BaseNet. The model consists of three main components: a shared encoder, a CTC decoder and attention decoders. The training process utilized a joint loss to optimize performance.

Fig. 6

Paraformer Architecture in BaseNet. The schematic depicts the architecture of the Paraformer model developed in BaseNet. The model features an encoder for generating hidden representations, a predictor for producing acoustic embeddings and predicting sequence lengths, a sampler for randomly creating semantic embeddings, and a decoder for generating outputs.

Fig. 7

Performance of BaseNet Under different learning rates and restricted weights. (a) Learning rate variation under three schedulers: CosineDecay, Noam, and WarmupLR. The Warmup step is set to 1000 and the total step to 10,000. (b) Autoregressive transformer prediction performance with different restricted weights. Performance significantly reduces with no EOS constraint (w=0) or strict constraints (w>=0.8). Optimal performance is observed within the range of w values from 0.1 to 0.7.

Fig. 8

Cross-attention weights between current signals and base sequences in transformer decoder. (a) Visualization of cross-attention weights across different decoder layers for a specific sequence. The linear relationship between signal and sequence follows a weak-strong-weak pattern among layers. Layers four to six exhibit the strongest linear relationships, indicating that they effectively capture both local and global features. In contrast, layers one to three and seven to eight show no or weaker linear relationships, indicating a focus on either local or global features without integrating both. (b) Visualization of cross-attention weights across different sequences in decoder layer four.

Fig. 9

Performance comparison of different large-scale models. (a) Performance comparison of different fine-tuned models. Among the 7 fine-tuned models, the model with one additional linear layer achieved the best performance, reaching 95.86 % identity after 10 epochs of fine-tuning. (b) Accuracy comparison of large models under different training conditions as indicated. (c) Training loss curves for signal fine-tuning, speech fine-tuning, and training from scratch.

Fig. 10

Performance comparison of different basecallers. (a) Identity comparison of different basecallers. (b) Identity comparison of different basecallers under different sequence lengths. Error bars represent standard errors. Fine-tuned model performs best across all lengths, and when the sequence length exceeds 20,000, the Joint-CTC model outperforms the Bonito-CRF model.