Benchmarking Deep Networks for Predicting Residue-Specific Quality of Individual Protein Models in CASP11

Abstract

Quality assessment of a protein model is to predict the absolute or relative quality of a protein model using computational methods before the native structure is available. Single-model methods only need one model as input and can predict the absolute residue-specific quality of an individual model. Here, we have developed four novel single-model methods (Wang_deep_1, Wang_deep_2, Wang_deep_3, and Wang_SVM) based on stacked denoising autoencoders (SdAs) and support vector machines (SVMs). We evaluated these four methods along with six other methods participating in CASP11 at the global and local levels using Pearson's correlation coefficients and ROC analysis. As for residue-specific quality assessment, our four methods achieved better performance than most of the six other CASP11 methods in distinguishing the reliably modeled residues from the unreliable measured by ROC analysis; and our SdA-based method Wang_deep_1 has achieved the highest accuracy, 0.77, compared to SVM-based methods and our ensemble of an SVM and SdAs. However, we found that Wang_deep_2 and Wang_deep_3, both based on an ensemble of multiple SdAs and an SVM, performed slightly better than Wang_deep_1 in terms of ROC analysis, indicating that integrating an SVM with deep networks works well in terms of certain measurements.

Figure 1. Local evaluation—the percentage of residues whose predicted deviations are within five different sets when the observed deviations of target residues belong to a range of (a) [0 Å, 2.5 Å), (b) [2.5 Å, 5 Å), (c) [5 Å, 7.5 Å), and (d) [7.5 Å, 10 Å), respectively.

Figure 2. Local evaluation—the residue-specific prediction assessment by weight mean PMCC of all models in the pool of targets of interest, TBM proteins, and FM proteins separately.

Figure 3. Local evaluation—the MCC and ACC when the threshold is set to 5 Å.

An estimate is considered correct when both predicted and observed deviations are within 5 Å.

Figure 4. Local evaluation—the ROC analysis in stage_2 to assess the ability to identify reliable residues from unreliable residues.

(a) The ROC curves for ten CASP11 QA tools and (b) the corresponding AUCs. Group names are sorted by their AUCs.

Figure 5

(a) an example illustrating the predicted deviations generated by Wang_deep_2 and the observed deviations from superimposing the experimental structure on the predicted model. The model in this example is “raghavagps-tsppred_TS3” for target T0819 in CASP11. (b) The visualization of the superimposition with the model in the color blue or red whereas red regions are the segments that have relatively larger real distances (>~3.5 Å) and blue for regions with relatively smaller real distances.

Figure 6. The architecture of stacked denoising autoencoders (SdAs).

Figure 7. The architecture of Wang_deep_3, in which the original input X was not only input into nine SdAs, but also the Support Vector Machine, together with the 20 predicted probabilities from each SdA.