Quantification of biases in predictions of protein-protein binding affinity changes upon mutations

Abstract

Understanding the impact of mutations on protein-protein binding affinity is a key objective for a wide range of biotechnological applications and for shedding light on disease-causing mutations, which are often located at protein-protein interfaces. Over the past decade, many computational methods using physics-based and/or machine learning approaches have been developed to predict how protein binding affinity changes upon mutations. They all claim to achieve astonishing accuracy on both training and test sets, with performances on standard benchmarks such as SKEMPI 2.0 that seem overly optimistic. Here we benchmarked eight well-known and well-used predictors and identified their biases and dataset dependencies, using not only SKEMPI 2.0 as a test set but also deep mutagenesis data on the severe acute respiratory syndrome coronavirus 2 spike protein in complex with the human angiotensin-converting enzyme 2. We showed that, even though most of the tested methods reach a significant degree of robustness and accuracy, they suffer from limited generalizability properties and struggle to predict unseen mutations. Interestingly, the generalizability problems are more severe for pure machine learning approaches, while physics-based methods are less affected by this issue. Moreover, undesirable prediction biases toward specific mutation properties, the most marked being toward destabilizing mutations, are also observed and should be carefully considered by method developers. We conclude from our analyses that there is room for improvement in the prediction models and suggest ways to check, assess and improve their generalizability and robustness.

Keywords: machine learning; prediction biases; protein complex structure; protein–protein binding affinity; protein–protein interactions; symmetry principle.

Figure 1

Characteristics of the S dataset. (A) Number of occurrences of mutation types; (B) Distribution of the experimental values (in kcal/mol).

Figure 2

Pearson correlations between experimental and predicted values on direct (in blue) and reverse (in orange) mutations of S (left) and C (right).

Figure 3

Predicted values as a function of experimental values (in kcal/mol) for the datasets S-D (blue dots) and S-R (orange dots). Predictions are obtained with mCSM-PPI2, MutaBind2, BeAtMuSiC, SSIPe, SAAMBE-3D, NetTree, BindProfX and FoldX.

Figure 4

Relation between the covering ratio and the Pearson correlation between predicted and experimental values on the S-D set for six benchmarked predictors. The linear regression line (dashed) and coefficient of determination () are indicated.

Figure 5

Distribution of the shift (in kcal/mol) for the eight benchmarked predictors calculated for mutations from C. The vertical blue dashed lines indicate and the vertical red dashed lines, the value of .

Figure 6

Normalized RMSE () of the eight predictors on subsets of S-D. Subsets were defined based on (a) mutation type: mutation toward Ala (A) versus other mutations (nA); (b) mutation location: mutations at the interface (I) versus other mutations (nI). (c) complex type: mutation on dimeric complexes (D) versus mutations on multi-n-meric complexes () (nD).