Assessment of protein model structure accuracy estimation in CASP13: Challenges in the era of deep learning

Abstract

Scoring model structure is an essential component of protein structure prediction that can affect the prediction accuracy tremendously. Users of protein structure prediction results also need to score models to select the best models for their application studies. In Critical Assessment of techniques for protein Structure Prediction (CASP), model accuracy estimation methods have been tested in a blind fashion by providing models submitted by the tertiary structure prediction servers for scoring. In CASP13, model accuracy estimation results were evaluated in terms of both global and local structure accuracy. Global structure accuracy estimation was evaluated by the quality of the models selected by the global structure scores and by the absolute estimates of the global scores. Residue-wise, local structure accuracy estimations were evaluated by three different measures. A new measure introduced in CASP13 evaluates the ability to predict inaccurately modeled regions that may be improved by refinement. An intensive comparative analysis on CASP13 and the previous CASPs revealed that the tertiary structure models generated by the CASP13 servers show very distinct features. Higher consensus toward models of higher global accuracy appeared even for free modeling targets, and many models of high global accuracy were not well optimized at the atomic level. This is related to the new technology in CASP13, deep learning for tertiary contact prediction. The tertiary model structures generated by deep learning pose a new challenge for EMA (estimation of model accuracy) method developers. Model accuracy estimation itself is also an area where deep learning can potentially have an impact, although current EMA methods have not fully explored that direction.

Keywords: CASP13 assessment; estimation of protein model accuracy; protein model quality assessment; protein structure prediction.

Figure 1.

Ranking of the methods in global accuracy estimation in terms of top 1 loss. (A) Sum of average Z-score of top 1 GDT-TS loss and that of top 1 LDDT loss used to rank the methods is shown for each group, single-model methods in green and consensus methods in black. (B) Average values of top 1 GDT-TS/LDDT loss are shown. Scores for GDT-TS are shown in red, and those for LDDT in blue.

Figure 2.

Performance of EMA methods in average top 1 GDT-TS loss when EMA methods are used to select models from TS servers. (A) Comparison with TS servers on all targets and (B) comparison with all TS groups on human targets. Single-model EMA methods are colored in green, consensus EMA methods in black, TS servers in orange, and TS human groups in red.

Figure 3.

Ranking of the methods in global accuracy estimation in absolute accuracy estimation. (A) Sum of average Z-score of GDT-TS error and that of LDDT error used to rank the methods is shown for each group, single-model methods in green and consensus methods in black. (B) Average values of the absolute GDT-TS/LDDT error are shown. Scores for GDT-TS are shown in red, and those for LDDT in blue.

Figure 4.

Ranking of the methods in local accuracy estimation. (A) Sum of average Z-scores for ULR (yellow), AUC (red), and ASE (blue) used to rank the methods is shown for each group, single-model methods in green and consensus methods in black. (B) Average values of the individual measures are shown.

Figure 5.

Performance comparison of EMA methods relative to the reference methods over the last three CASPs. (A) Ratio of top 1 loss of ‘Davis-EMAconsensus’ to that of the best consensus EMA method and (B) Ratio of top 1 loss of ‘GOAP’ to that of the best single-model method.