Assessment of protein disorder region predictions in CASP10

Abstract

The article presents the assessment of disorder region predictions submitted to CASP10. The evaluation is based on the three measures tested in previous CASPs: (i) balanced accuracy, (ii) the Matthews correlation coefficient for the binary predictions, and (iii) the area under the curve in the receiver operating characteristic (ROC) analysis of predictions using probability annotation. We also performed new analyses such as comparison of the submitted predictions with those obtained with a Naïve disorder prediction method and with predictions from the disorder prediction databases D2P2 and MobiDB. On average, the methods participating in CASP10 demonstrated slightly better performance than those in CASP9.

Keywords: CASP; assessment of disorder prediction; intrinsically disordered proteins; prediction of disordered regions; unstructured proteins.

Figure 1

Distribution of disordered regions of different lengths in CASP10 targets. Internal regions are shown with darker shades.

Figure 2

Fraction of disordered residues at the termini of CASP10 targets. L is the target length (in number of residues).

Figure 3

Performance of groups as binary disorder/order classifiers according to the MCC, Acc, and precision scores. The error bars indicate boundaries of the 95% confidence intervals for each measure. Groups are ordered according to decreasing MCC score.

Figure 4

(A) ROC and (B) PR curves for the probability-based disorder region predictions for all CASP10 groups. The three groups with the atypical ROC curves in (A) used only a very limited number of disorder probability values: Groups G167 and G168 used only two different values (one for ordered and another for disordered residues); Group G115 used only five different values (two for ordered and three for disordered residues, and two out of the five values were only assigned to a very small number of residues). Groups in the legend are sorted according to decreasing AUC_ROC score.

Figure 5

Comparison of group performance on the full-length and termini-trimmed targets according to the (A) MCC and (B) AUC scores. Scores on the trimmed targets are marked as “internal.”

Figure 6

Comparison of group performance for four different thresholds of the minimum length of disordered regions. The two panels show data for two different evaluation measures (MCC and AUC). Each group is shown with a different color; groups in the legend are sorted according to the AUC score (across and then down); the artificial average group (“AVRG,” black thicker line) is added to the graph for reference.

Figure 7

Comparison of performance for the best 12 groups in the last four CASPs according to the MCC (A) and AUC (B) scores. Groups in each panel and are sorted according to decreasing scores in each CASP. CASP8* scores are calculated without a very long and completely unstructured target T0500, the correct prediction of which would over-inflate the evaluation scores.