Assessment of ligand binding residue predictions in CASP8

Abstract

Here we detail the assessment process for the binding site prediction category of the eighth Critical Assessment of Protein Structure Prediction experiment (CASP8). Predictions were only evaluated for those targets that bound biologically relevant ligands and were assessed using the Matthews Correlation Coefficient. The results of the analysis clearly demonstrate that three predictors from two groups (Lee and Sternberg) stand out from the rest. A further two groups perform well over subsets of metal binding or nonmetal ligand binding targets. The best methods were able to make consistently reliable predictions based on model structures, though it was noticeable that the two targets that were not well predicted were also the hardest targets. The number of predictors that submitted new methods in this category was highly encouraging and suggests that current technology is at the level that experimental biochemists and structural biologists could benefit from what is clearly a growing field.

Fig. 1. Overview of predictions per group

Predictions for targets which were assessed in the FN category (i.e. targets with a relevant binding site) are displayed in dark colors, additional predictions which were not assessed (i.e. targets without an experimentally confirmed binding site) are displayed in light colors. Human groups are shown in purple, servers in orange.

Fig. 2. Mean Z scores over all targets for the top 20 predictor groups

Error bars show minimum and maximum average Z scores obtained from bootstrapping experiment. Human predictor groups are shown in purple, servers in orange.

Fig. 3. Mean rank based on bootstrapping experiment for the top 20 predictor groups

Error bars show minimum and maximum rank obtained from bootstrapping experiment. Human predictors are shown in purple, servers in orange.

Fig. 4. Mean Z scores of the top 20 groups, separated by the ligand’s chemotype

Metals are shown in blue, non-metals are shown in green.

Fig. 5. MCC scores for the 12 top performing groups for all targets

Targets were sorted by their respective MCC score, individually for each group.

Fig. 6. Overall target difficulty

MCC value of the best overall prediction for each target.

Fig. 7. Number of targets where a particular group returned the best prediction

Groups are sorted by their overall performance. For one target, multiple groups can perform equally.

Fig. 8. Examples of binding site predictions

All ligands are shown in spheres render mode. The protein backbone is shown in cartoon mode with each chain colored separately. All side chains of observed and predicted binding site residues are shown in licorice sticks. Correctly predicted residues (true positives) are colored in green, incorrectly under predicted binding site residues (false negatives) in yellow and incorrectly over predicted non-binding site residues (false positives) in red. (A) Target T0604 with predictions of group FN035. (B) Predictions of group FN096 for target T0632. (C) Group FN114’s predictions for target T0629.