Comparative analysis of methods for evaluation of protein models against native structures

Abstract

Motivation: Measuring discrepancies between protein models and native structures is at the heart of development of protein structure prediction methods and comparison of their performance. A number of different evaluation methods have been developed; however, their comprehensive and unbiased comparison has not been performed.

Results: We carried out a comparative analysis of several popular model assessment methods (RMSD, TM-score, GDT, QCS, CAD-score, LDDT, SphereGrinder and RPF) to reveal their relative strengths and weaknesses. The analysis, performed on a large and diverse model set derived in the course of three latest community-wide CASP experiments (CASP10-12), had two major directions. First, we looked at general differences between the scores by analyzing distribution, correspondence and correlation of their values as well as differences in selecting best models. Second, we examined the score differences taking into account various structural properties of models (stereochemistry, hydrogen bonds, packing of domains and chain fragments, missing residues, protein length and secondary structure). Our results provide a solid basis for an informed selection of the most appropriate score or combination of scores depending on the task at hand.

Supplementary information: Supplementary data are available at Bioinformatics online.

Fig. 1.

Empirical distribution of score values on the single-domain dataset. Horizontal axis indicates score values, vertical axis—frequency of the value occurrence. For RMSD/tRMSD, GDT and CAD-score only representative versions are shown [Cα RMSD/tRMSD, GDT-TS and all atom CAD-score (CAD-AA)]

Fig. 2.

Correspondence of scores. Scatter plots for ‘A’ and ‘B’ score pairs. Horizontal direction represents values of score ‘A’, vertical direction represents score ‘B’. Increasing color intensity represents the increasing local density of values

Fig. 3.

Correlation of scores. (A) Spearman’s rank correlation coefficients computed by averaging per target values. Coloring ranges from blue (high correlation) to red (low correlation). (B) Clustering of scores according to their correlations using multi-dimensional scaling (MDS) (Color version of this figure is available at Bioinformatics online.)

Fig. 4.

Score differences in selecting a better model out of two. (A) Fractions of model pairs, where the disagreement between scores exceeds the tolerance threshold. Differences are colored from blue (smallest) to red (largest). (B) Clustering of scores based on the analysis of model pairs with conflicting ranking (Color version of this figure is available at Bioinformatics online.)

Fig. 5.

Agreement between the scores in selecting the best model out of many. (A) Average fraction of the same selections. (B) Clustering of scores according to their agreement in selecting the best model

Fig. 6.

Score differences in selecting the best model out of many. (A) Average losses of Z-score computed by score B (vertical axis) when the best model is selected using score A (horizontal axis). Z-score losses are colored from blue (smallest) to red (largest). Note the overall asymmetry of Z-score loss for score pairs. (B) Clustering of scores according to average Z-score losses (Color version of this figure is available at Bioinformatics online.)

Fig. 7.

Conflicting rankings of model pairs ‘judged’ using MolProbity. Fractions of conflicting model pairs for which MolProbity supports score ‘A’ (vertical axis) over score ‘B’ (horizontal axis). The MolProbity support is colored from blue (largest) to red (smallest) (Color version of this figure is available at Bioinformatics online.)

Fig. 8.

Conflicting rankings of model pairs ‘judged’ by the number of reproduced hydrogen bonds. The scores are compared using differences larger than a defined threshold (A) for all hydrogen bonds and (B) for only non-local hydrogen bonds

Fig. 9.

Grishin plots reflecting sensitivity of different scores to the relative orientation of protein domains. Horizontal axis indicates the score values for the full structure, vertical axis indicates weighted sum of scores for individual domains

Fig. 10.

Different types of local structural deviations. CASP10 target T0663-D1 (blue) superimposed with three models: (A) TS301_5 (orange), (B) TS301_3 (yellow) and (C) TS476_4 (grey). Red arrow indicates the C-terminal region featuring different types of deviation in the models (Color version of this figure is available at Bioinformatics online.)

Fig. 11.

Response of the scores to model completeness. Horizontal axis indicates the model completeness as the percentage of residues modeled. Vertical axis indicates mean Z-score for models of the same degree of completeness. Z-scores are averaged using left-sided sliding window of 15%, e.g. values at 80% are averages of 95–80% range