Machine learning for RNA 2D structure prediction benchmarked on experimental data

Abstract

Since the 1980s, dozens of computational methods have addressed the problem of predicting RNA secondary structure. Among them are those that follow standard optimization approaches and, more recently, machine learning (ML) algorithms. The former were repeatedly benchmarked on various datasets. The latter, on the other hand, have not yet undergone extensive analysis that could suggest to the user which algorithm best fits the problem to be solved. In this review, we compare 15 methods that predict the secondary structure of RNA, of which 6 are based on deep learning (DL), 3 on shallow learning (SL) and 6 control methods on non-ML approaches. We discuss the ML strategies implemented and perform three experiments in which we evaluate the prediction of (I) representatives of the RNA equivalence classes, (II) selected Rfam sequences and (III) RNAs from new Rfam families. We show that DL-based algorithms (such as SPOT-RNA and UFold) can outperform SL and traditional methods if the data distribution is similar in the training and testing set. However, when predicting 2D structures for new RNA families, the advantage of DL is no longer clear, and its performance is inferior or equal to that of SL and non-ML methods.

Keywords: RNA 2D structure prediction; algorithm benchmarking; deep learning; machine learning.

Figure 1

Interaction Network Fidelity (INF) computed for canonical base pairs, in Experiment I (predicting representatives of equivalence classes). Colors refer to groups of algorithms: blue – deep learning (DL), orange – shallow learning (SL) and green – non-ML algorithms.

Figure 2

Interaction Network Fidelity (INF) computed for canonical base pairs, in Experiment II (predicting selected Rfam sequences). Colors refer to groups of algorithms: blue – deep learning (DL), orange – shallow learning (SL) and green – non-ML algorithms.

Figure 3

Interaction Network Fidelity (INF) computed for canonical base pairs, in Experiment III (predicting new Rfam families). Colors refer to groups of algorithms: blue – deep learning (DL), orange – shallow learning (SL) and green – non-ML algorithms.