The landscape of RNA 3D structure modeling with transformer networks

Abstract

Transformers are a powerful subclass of neural networks catalyzing the development of a growing number of computational methods for RNA structure modeling. Here, we conduct an objective and empirical study of the predictive modeling accuracy of the emerging transformer-based methods for RNA structure prediction. Our study reveals multi-faceted complementarity between the methods and underscores some key aspects that affect the prediction accuracy.

Figure 1.

Partial head-to-head comparison of five transformer-based RNA 3D modeling methods on a benchmark set of 72 targets (The full set of comparisons is available in Supplementary Figs S1–S3). Scatterplots showing comparisons between two competing methods. Percentage numbers reported in the top-left quadrant correspond to targets where the method shown in the Y-axis outperforms the competing method shown in the X-axis, and vice versa for the bottom-right quadrant. The distributions of the scores color-coded for different methods for ease of visualization are demonstrated for both axes, with the dashed line and the numbers next to it indicating the mean values.

Figure 2.

Results of pairwise statistical tests (p-values) of the performance difference between all five deep learning-based methods in terms of three different performance evaluation metrics for three partitions of the benchmark set: (i) CASP15 set having only 12 RNA targets (CASP15); (ii) our in-house benchmarking set of 60 non-redundant RNA targets (TS60); and (iii) the combination of the two sets having a total of 72 RNA targets (TS72).

Figure 3.

Predictive modeling accuracy of five transformer-based RNA 3D modeling methods on a benchmark set of 72 targets. (a) Four representative targets shown with the predicted structural models colored in blue superimposed on the experimental structures in green and the TM-score and INF scores shown below with bold numbers indicating the best performance. (b) Polar bar plot showing the mean accuracies of the methods annotated on top of each bar along with the impact in accuracy with the change in sequence length (L). (c) Notched box plots showing the TM-score distributions with and without using MSA as input for the MSA-based methods with numbers indicating the median values (top); and scatterplot between predicted TM-scores and MSA depths (log(N_eff) (bottom) where the solid lines represent tendency lines constructed by linear fit to the data. (d) Joint angular distribution of the pseudo torsion angles, with color code ramping from blue to red for low to high density.