BBANsh: a deep learning architecture based on BERT and bilinear attention networks to identify potent shRNA

Abstract

RNA interference (RNAi) is a technique for precisely silencing the expression of specific genes by means of small RNA molecules and is essential in functional genomics. Among the commonly used RNAi molecules, short hairpin RNAs (shRNAs) exhibit advantages over small interfering RNAs, including longer half-life, comparable silencing efficiency, fewer off-target effects, and greater safety. However, traditional screening of potent shRNAs is costly and time-consuming. Advances in big data and artificial intelligence have enabled computational methods to significantly accelerate shRNA design and prediction. In this study, we propose BBANsh, a new shRNA prediction model based on bidirectional encoder representation from transformers (BERT) and bilinear attention network (BAN). We comprehensively evaluate the performance of BBANsh against traditional feature-based models, various feature fusion methods, and existing shRNA prediction models. The BBANsh has achieved an area under the precision-recall curve of 0.951 on five-cross validation and a prediction accuracy of 0.896 on a new external validation set, highlighting its superior predictive performance. Ablation experiments validate the significant contributions of BERT and BAN to model performance. The visualization of internal feature representations intuitively demonstrates the effectiveness of the feature fusion strategy of BBANsh. Furthermore, the attentional analysis reveals that nucleotides near the 5' end have the greatest impact on model predictions, highlighting sequence characteristics of potent shRNAs. Overall, BBANsh provides an efficient and reliable tool for shRNA prediction, which can offer valuable support for researchers in the precise selection and design of shRNA.

Keywords: BERT; bilinear attention network; shRNA prediction.

Figure 1

The workflow of BBANsh. (a) Collection of shRNA dataset: data were collected from the TELE and M1 datasets, both of which were generated using sensor-based experimental systems. (b) Encoding process using BERT for shRNA: a pretrained BERT model was fine-tuned on the shRNA dataset, and the final [CLS] token embeddings were used to represent each shRNA sequence. (c) Feature fusion with bilinear attention network: the embeddings derived from DNABERT and GENA-LM, denoted as and , respectively, were fused using a bilinear attention network. (d) The complete framework of BBANsh: an overview of the BBANsh architecture, integrating the encoding, fusion, and prediction components.

Figure 2

The performance of five-fold cross-validation and test validation for various models and BBANsh. (a) the comparison of BBANsh and traditional feature-based models, including kmer (k = 3), PseDNC, and PseKNC. (b) The comparison of BBANsh with other models utilizing different feature fusion strategies, including concatenation (Concat) and Co-attention mechanisms.

Figure 3

The performance of BBANsh and ILGBMSH using leave-one-gene-out cross-validation across nine genes. (a) Receiver operating characteristic (ROC) curves for each model on all nine genes. (b) Precision–recall (PR) curves illustrating the precision versus recall performance for BBANsh and ILGBMSH.

Figure 4

The ablation results of BBANsh variants on five-fold cross-validation and test set validation. Model variants are evaluated by removing key components: (i) noBAN: replacing the bilinear attention network with simple concatenation, (ii) noGENA-LM: removing the GENA-LM embedding, and (iii) noDNABERT: removing the DNABERT embedding.

Figure 5

T-SNE visualization of feature layers of the BBANsh model (including features retrieved from joint features post-BAN fusion, 128-dimensional MLP layer features, and 64-dimensional MLP layer features), models constructed using traditional features (including kmer, PseDNC, and PseKNC), DNABERT, GENA-LM, and models employing various fusion methodologies (Concat and Co-attention).

Figure 6

Visualization of the attention weights on sequence features in the BBANsh model, highlighting which nucleotide positions the model attends to most during prediction.