Comparative evaluation of AlphaFold2 and disorder predictors for prediction of intrinsic disorder, disorder content and fully disordered proteins

Abstract

We expand studies of AlphaFold2 (AF2) in the context of intrinsic disorder prediction by comparing it against a broad selection of 20 accurate, popular and recently released disorder predictors. We use 25% larger benchmark dataset with 646 proteins and cover protein-level predictions of disorder content and fully disordered proteins. AF2-based disorder predictions secure a relatively high Area Under receiver operating characteristic Curve (AUC) of 0.77 and are statistically outperformed by several modern disorder predictors that secure AUCs around 0.8 with median runtime of about 20 s compared to 1200 s for AF2. Moreover, AF2 provides modestly accurate predictions of fully disordered proteins (F1 = 0.59 vs. 0.91 for the best disorder predictor) and disorder content (mean absolute error of 0.21 vs. 0.15). AF2 also generates statistically more accurate disorder predictions for about 20% of proteins that have relatively short sequences and a few disordered regions that tend to be located at the sequence termini, and which are absent of disordered protein-binding regions. Interestingly, AF2 and the most accurate disorder predictors rely on deep neural networks, suggesting that these models are useful for protein structure and disorder predictions.

Keywords: AlphaFold2; Deep learning; Disorder content; Fully disordered proteins; Intrinsic disorder; Intrinsically disordered protein; Prediction.

Graphical abstract

Fig. 1

Illustrative examples of proteins that represent the four types of intrinsically disordered proteins: shortIDR (i.e., have only short IDRs), longIDR (have at least one long IDR), bindingIDR (have at least one binding IDR); and non-terminusIDR (do not have IDRs at the sequence termini). We identify proteins by their DisProt and UniProt identifiers. We draw IDRs as brown (for non-binding) and purple (for binding) segments.

Fig. 2

Comparison of the sequence-derived markers between proteins for which AF2-RSA generates competitive predictions (blue box plots) vs. proteins for which AF2-RSA is statistically outperformed by disorder predictions or generates lower accuracy predictions (red box plots). The markers include: (panel a) sequence length; (panel b) putative content of binding IDRs; (panel c) number of putative IDRs; (panel d) composite score of distance to terminus and content of the putative disorder; (panel e) distance of putative IDRs to a closest terminus; (panel f) putative disorder content; (panel g) maximal length of putative IDRs; and (panel h) putative content of coiled-coils. Box plots represent distributions of the marker values in a given protein set where we show the 5th, 25th, 50th (median), 75th and 95th percentiles and where cross represents the average. Statistical significance of differences is annotated above the box plots: ns means difference is not significant; * means significant.