CASP9 assessment of free modeling target predictions

Abstract

We present an overview of the ninth round of Critical Assessment of Protein Structure Prediction (CASP9) "Template free modeling" category (FM). Prediction models were evaluated using a combination of established structural and sequence comparison measures and a novel automated method designed to mimic manual inspection by capturing both global and local structural features. These scores were compared to those assigned manually over a diverse subset of target domains. Scores were combined to compare overall performance of participating groups and to estimate rank significance. Moreover, we discuss a few examples of free modeling targets to highlight the progress and bottlenecks of current prediction methods. Notably, a server prediction model for a single target (T0581) improved significantly over the closest structure template (44% GDT increase). This accomplishment represents the "winner" of the CASP9 FM category. A number of human expert groups submitted slight variations of this model, highlighting a trend for human experts to act as "meta predictors" by correctly selecting among models produced by the top-performing automated servers. The details of evaluation are available at http://prodata.swmed.edu/CASP9/ .

Figure 1. Manual FM Target Evaluation Criteria

A diverse subset of FM targets (15 individual domains and 1 combined domain) provided the basis for manual evaluation. For each target, important secondary structures were colored, with connecting loops left white. Key interaction residues were highlighted with magenta sticks to evaluate the relative positions of secondary structures. These criteria provided the basis for side-by-side comparisons to all prediction models.

Figure 2. Target 531 Manual Top-Scoring Predictions

(A) Target 531 is colored with secondary structure criteria as in Figure 1. Key contact residues highlight the orientation and shift of the β-strand meander (black) and special structural features specific to this target (magenta disulfide bonds). (B) Rank 1 prediction model (399_4) is colored in rainbow and illustrates the correct features of the β-strand meander (black residues) and the relative placement of the flanking helices. (C) Rank 2 prediction model (55_1) is colored as in B and includes a better prediction for the second helix (green), but a shifted second β-strand. (D) Top ranked GDT model (399_4) is colored as in B and is missing a key feature of the core β-meander (cyan strand). (E) Correlation plot of all target 531 non-zero manual scores with GDT scores shows a positive correlation (R=0.53), albeit with different ranking of top prediction models (labeled according to illustrated examples).

Figure 3. QCS reveals good structural features

(A) QCS selects the best prediction model TS382_5 for (B) the target T0561, (C) while GDT selects a different prediction, TS324_5, as the best model. Topology diagrams are displayed for (D) TS382_5, (E) T0561, and (F) TS324_5. Identified SSE interactions are colored (magenta) (G) for QCS selected model TS382_5, (H) for target T0561, and for (I) GDT selected model TS324_5. (J) Target 550d2 forms an 8-stranded β-meander barrel with longer C-terminal 3 helices. (K) QCS favored model TS408_2 forms a β-meander barrel with an extra strand (gray), while (L) TenS favored model TS428_4 forms a similarly shaped barrel with C-terminal strands of similar length but an altered topology (gray).

Figure 4. Group Performance

(A) Average best model ComS Z-scores (black diamonds) for the top ten ranked of all groups on FM domains designated as human/server are compared to Average ComS Z-scores on models designated as first (gray diamonds). (B) Average best ComS Z-scores (black diamonds) for the top ten ranked of Server-only groups on FM domains designated as Server-only are compared to Average ComS Z-scores on models designated as first (gray diamonds). (C) Column graph illustrates best Server ratio scores (indicated above column) summed over all FM domains designated as human/server. Groups (96, 172, 470, and 408) that outperformed top servers (380, 321, and 428) overall are included in the graph.

Figure 5. Top Models Compared to Top Template Performance

The top ten best prediction model GDT scores (black circles) for each FM domain (labeled below) are plotted together with the GDT score for the best available template (white triangles). Target domains are ordered according to their similarity with the closest template.

Figure 6. Highlights and Pitfalls of FM Predictions

(A) The best server prediction (321_4) for (B) the target T0581 is compared to (C) closest template structure represented by the C-terminal domain of Fatty acyl-AMP ligase (3lnv). (D) The best server prediction (428_1) for (E) the target T0604_1 is compared to (F) the closest template structure represented by the structure of human LINE-1 ORF1P central domain (2w7a). (G) Target T0534 has discontinuous domain bounds resulting from a four helix up-and-down bundle (red) inserted into another all-helical bromodomain-like fold (blue). Domain organization is mapped to sequence below. (H) T0534_1 includes a 3 helical insertion (salmon) in the center of one helix (orange) that is captured by (I) a prediction model (172_5). Domain organization is mapped to sequence below. (J) Target T0550_2β-sheet displays typical backbone hydrogen bonds between adjacent β-strands. (K) A top server prediction model (428_4) for T0550_2 displays incorrect β-sheet hydrogen-bonding patterns. (L) Another server model (2_5) has steric clashes between loop backbones and sidechains.

Figure 7. CASP9 Performance Compared to CASP8 by Sam-T08 Server

(A) Distribution of SAM-T08 GDT scores for the defined CASP9 FM targets (gray bars) suggests CASP8 targets with SAM-T08 GDT scores below 46 are matched in difficulty (black bars). Bars representing the most difficult CASP9 targets are labeled by domain name. (B) Distribution of best model GDT/SAM-T08 GDT ratio scores for all FM targets in CASP9 (gray bars) and for matched targets in CASP8 (black bars). Bars corresponding to Top-performing CASP9 target domains are labeled.