All are not equal: a benchmark of different homology modeling programs

Abstract

Modeling a protein structure based on a homologous structure is a standard method in structural biology today. In this process an alignment of a target protein sequence onto the structure of a template(s) is used as input to a program that constructs a 3D model. It has been shown that the most important factor in this process is the correctness of the alignment and the choice of the best template structure(s), while it is generally believed that there are no major differences between the best modeling programs. Therefore, a large number of studies to benchmark the alignment qualities and the selection process have been performed. However, to our knowledge no large-scale benchmark has been performed to evaluate the programs used to transform the alignment to a 3D model. In this study, a benchmark of six different homology modeling programs- Modeller, SegMod/ENCAD, SWISS-MODEL, 3D-JIGSAW, nest, and Builder-is presented. The performance of these programs is evaluated using physiochemical correctness and structural similarity to the correct structure. From our analysis it can be concluded that no single modeling program outperform the others in all tests. However, it is quite clear that three modeling programs, Modeller, nest, and SegMod/ ENCAD, perform better than the others. Interestingly, the fastest and oldest modeling program, SegMod/ ENCAD, performs very well, although it was written more than 10 years ago and has not undergone any development since. It can also be observed that none of the homology modeling programs builds side chains as well as a specialized program (SCWRL), and therefore there should be room for improvement.

Figure 1.

Example of models produced with an alignment containing an error. (A) SWISS-MODEL model, (B) Backbone model, (C) Modeller model, (D) Native structure. The N-terminal helix in the backbone model is clearly wrong, since the distance between two adjacent residues is 40 Å. For the Modeller model this has no great impact (it is just one of many restraints); however, for the SWISS-MODEL model the error is enough to break the sheet in order to include the helix in the model. Figures were made using MOLSCRIPT (Kraulis 1991).

Figure 2.

Histogram over the number of models that contain missing residues, i.e., where the program for some reason does not model all residues in the target sequence. SCWRL3 does not attempt to model loops; therefore, this number represents the alignments containing gaps.

Figure 3.

Different measures used to assess the quality of the protein models. (A) RMSD values transformed using 1/(1 + RMSD) to avoid problem with high values. (B) MaxSub. (C) Backbone quality. (D) Side-chain quality as measured by fraction of correct side-chain torsion angles (χ₁ and χ₂). Error bars are constructed using standard error.

Figure 4.

(A) Fraction of residues with “bad” bond angles, bond length side-chain planarity according to WHAT_CHECK for each method and also for the native structure. For a residue to be part of the any category it has to be classified as “bad” for any of the categories above. (B) The sequence identity dependence for the residues from the any “bad” category above. (C) Models with MaxSub score >0.6. (D) Acceptable model are models that have a MaxSub score of at least 0.6 and not more than 10% of its residues missing or with bad chemistry. 3D-JIGSAW and Builder have a significantly lower number of acceptable models and were removed for clarity.

Figure 5.

Improvement over template as measured by (A) the difference between the fraction of models that gets significantly (ΔMX > 0.02) improved, f_imp, and the fraction that gets significantly deteriorated, f_det, or by (B) the average fraction of models that gets improved, 〈f_imp〉, and deteriorated, 〈f_det〉. 3D-JIGSAW and Builder were removed for clarity.

Figure 6.

Two examples of a model that is improved upon modeling: in red, the template structure model is shown; in green, the final model; in blue, the native structure. (A) Modeller7v7 model of domain d1heia2 with d8ohm 2 as a template. The alignment contains no gaps, and the sequence identity between the target and template sequence is 91%. The RMSD between the template and the native structure is 2.06 Å, and for the Modeller7v7 model only 1.40 Å. The MaxSub score is also improved from 0.75 to 0.87. (B) 3D-JIGSAW model of domain d1unka with d1emva as a template. The alignment contains a single residue gap, and the sequence identity between target and template sequence is 58%. The RMSD between the template and the native structure is 2.05 Å, and for the 3D-JIGSAW model 1.30 Å.The MaxSub score is improved from 0.80 to 0.87. Figures were made using MOLSCRIPT (Kraulis 1991).