Exploring QSAR models for activity-cliff prediction

Abstract

Introduction and methodology: Pairs of similar compounds that only differ by a small structural modification but exhibit a large difference in their binding affinity for a given target are known as activity cliffs (ACs). It has been hypothesised that QSAR models struggle to predict ACs and that ACs thus form a major source of prediction error. However, the AC-prediction power of modern QSAR methods and its quantitative relationship to general QSAR-prediction performance is still underexplored. We systematically construct nine distinct QSAR models by combining three molecular representation methods (extended-connectivity fingerprints, physicochemical-descriptor vectors and graph isomorphism networks) with three regression techniques (random forests, k-nearest neighbours and multilayer perceptrons); we then use each resulting model to classify pairs of similar compounds as ACs or non-ACs and to predict the activities of individual molecules in three case studies: dopamine receptor D2, factor Xa, and SARS-CoV-2 main protease.

Results and conclusions: Our results provide strong support for the hypothesis that indeed QSAR models frequently fail to predict ACs. We observe low AC-sensitivity amongst the evaluated models when the activities of both compounds are unknown, but a substantial increase in AC-sensitivity when the actual activity of one of the compounds is given. Graph isomorphism features are found to be competitive with or superior to classical molecular representations for AC-classification and can thus be employed as baseline AC-prediction models or simple compound-optimisation tools. For general QSAR-prediction, however, extended-connectivity fingerprints still consistently deliver the best performance amongs the tested input representations. A potential future pathway to improve QSAR-modelling performance might be the development of techniques to increase AC-sensitivity.

Keywords: Activity cliff prediction; Activity cliffs; Binding affinity prediction; Deep learning; Extended-connectivity fingerprints; Graph isomorphism networks; Machine learning; Molecular representation; Physicochemical descriptors; QSAR modelling.

Fig. 1

Example of an activity cliff (AC) for blood coagulation factor Xa. A small structural transformation in the upper compound leads to an increase in inhibitory activity of almost three orders of magnitude. Both compounds were identified in the same ChEMBL assay with ID 658338

Fig. 2

Illustration of our data splitting strategy. We distinguish between three MMP-sets, ${\mathcal{M}}_{\text{train}},{\mathcal{M}}_{\text{inter}}$ and ${\mathcal{M}}_{\text{test}}$, depending on whether both MMP-compounds are in ${\mathcal{D}}_{\text{train}}$, one MMP-compound is in ${\mathcal{D}}_{\text{train}}$ and the other one is in ${\mathcal{D}}_{\text{test}}$, or both MMP-compounds are in ${\mathcal{D}}_{\text{test}}$. We additionally consider a fourth MMP-set, ${\mathcal{M}}_{\text{cores}}$, consisting of the MMPs in ${\mathcal{M}}_{\text{test}}$ whose structural cores do not appear in ${\mathcal{M}}_{\text{train}}\cup {\mathcal{M}}_{\text{inter}}$

Fig. 3

Schematic showing the combinatorial experimental methodology used for the study. Each molecular representation method is systematically combined with each regression technique, giving a total of nine QSAR models. Each QSAR model is trained and evaluated for QSAR-prediction, AC-classification and PD-classification within a 2-fold cross validation scheme repeated with 3 random seeds. For each of the $2\ast 3=6$ trials, an extensive inner hyperparameter-optimisation loop on the training set is performed for each QSAR model

Fig. 4

QSAR-prediction- and AC-classification results for dopamine receptor D2. For each plot, the x-axis corresponds to a combination of MMP-set and AC-classification performance metric and the y-axis shows the QSAR-prediction performance on the molecular test set ${\mathcal{D}}_{\text{test}}$. The total length of each error bar equals twice the standard deviation of the performance metric measured over all $mk=3\ast 2=6$ hyperparameter-optimised models. For each plot, the lower right corner corresponds to strong performance at both prediction tasks

Fig. 5

QSAR-prediction- and AC-classification results for factor Xa. For each plot, the x-axis corresponds to a combination of MMP-set and AC-classification performance metric and the y-axis shows the QSAR-prediction performance on the molecular test set ${\mathcal{D}}_{\text{test}}$. The total length of each error bar equals twice the standard deviation of the performance metric measured over all $mk=3\ast 2=6$ hyperparameter-optimised models. For each plot, the lower right corner corresponds to strong performance at both prediction tasks

Fig. 6

QSAR-prediction- and AC-classification results for SARS CoV-2 main protease. For each plot, the x-axis corresponds to a combination of MMP-set and AC-classification performance metric and the y-axis shows the QSAR-prediction performance on the molecular test set ${\mathcal{D}}_{\text{test}}$. The total length of each error bar equals twice the standard deviation of the performance metric measured over all $mk=3\ast 2=6$ hyperparameter-optimised models. The precision of the AC-classification task is lacking for the ECFP + kNN technique on ${\mathcal{M}}_{\text{test}}$ and ${\mathcal{M}}_{\text{cores}}$ since this method produced only negative AC-predictions for all trials on this data set. For each plot, the lower right corner corresponds to strong performance at both prediction tasks

Fig. 7

QSAR-prediction- and PD-classification results for dopamine receptor D2. Each column corresponds to an upper plot and a lower plot for one of the MMP-sets ${\mathcal{M}}_{\text{inter}}$, ${\mathcal{M}}_{\text{test}}$ or ${\mathcal{M}}_{\text{cores}}$. The x-axis of each upper plot indicates the PD-classification accuracy on the full MMP-set; the x-axis of each lower plot indicates the PD-classification accuracy on a restricted MMP-set only consisting of MMP predicted to be ACs by the respective method. The y-axis of each plot shows the QSAR-prediction performance on the molecular test set ${\mathcal{D}}_{\text{test}}$. The total length of each error bar equals twice the standard deviation of the performance metrics measured over all $mk=3\ast 2=6$ hyperparameter-optimised models. For each plot, the lower right corner corresponds to strong performance at both prediction tasks

Fig. 8

QSAR-prediction- and PD-classification results for factor Xa. Each column corresponds to an upper plot and a lower plot for one of the MMP-sets ${\mathcal{M}}_{\text{inter}}$, ${\mathcal{M}}_{\text{test}}$ or ${\mathcal{M}}_{\text{cores}}$. The x-axis of each upper plot indicates the PD-classification accuracy on the full MMP-set; the x-axis of each lower plot indicates the PD-classification accuracy on a restricted MMP-set only consisting of MMP predicted to be ACs by the respective method. The y-axis of each plot shows the QSAR-prediction performance on the molecular test set ${\mathcal{D}}_{\text{test}}$. The total length of each error bar equals twice the standard deviation of the performance metrics measured over all $mk=3\ast 2=6$ hyperparameter-optimised models. For each plot, the lower right corner corresponds to strong performance at both prediction tasks

Fig. 9

QSAR-prediction- and PD-classification results for SARS-CoV-2 main protease. Each column corresponds to an upper plot and a lower plot for one of the MMP-sets ${\mathcal{M}}_{\text{inter}}$, ${\mathcal{M}}_{\text{test}}$ or ${\mathcal{M}}_{\text{cores}}$. The x-axis of each upper plot indicates the PD-classification accuracy on the full MMP-set; the x-axis of each lower plot indicates the PD-classification accuracy on a restricted MMP-set only consisting of MMP predicted to be ACs by the respective method. The y-axis of each plot shows the QSAR-prediction performance on the molecular test set ${\mathcal{D}}_{\text{test}}$. The total length of each error bar equals twice the standard deviation of the performance metrics measured over all $mk=3\ast 2=6$ hyperparameter-optimised models. The accuracy of the PD-classification task for predicted ACs is lacking for the ECFP + kNN technique on ${\mathcal{M}}_{\text{test}}$ and ${\mathcal{M}}_{\text{cores}}$ since this method produced only negative AC-predictions for all trials on this data set. For each plot, the lower right corner corresponds to strong performance at both prediction tasks