Improved Machine Learning Predictions of EC50s Using Uncertainty Estimation from Dose-Response Data

Abstract

In early-stage drug design, machine learning models often rely on compressed representations of data, where raw experimental results are distilled into a single metric per molecule through curve fitting. This process discards valuable information about the quality of the curve fit. In this study, we incorporated a fit-quality metric into machine learning models to capture the reliability of metrics for individual molecules. Using 40 data sets from PubChem (public) and BASF (private), we demonstrated that including this quality metric can significantly improve predictive performance without additional experiments. Four methods were tested: random forests with parametric bootstrap, weighted random forests, variable output smearing random forests, and weighted support vector regression. When using fit-quality metrics, at least one of these methods led to a statistically significant improvement on 31 of the 40 data sets. In the best case, these methods led to a 22% reduction in the root-mean-squared error of the models. Overall, our results demonstrate that by adapting data processing to account for curve fit quality, we can improve predictive performance across a range of different data sets.

1

(a) Reliable vs (b) unreliable fit of the Hill equation to experimental data points.

2

Graphical examples of regression and Bayesian fitting procedures.

3

Schematic comparing the standard approach to our modified testing procedure. The difference is in which values are tested: in this study we test how well we can predict experimental values rather than estimated EC50 values. The point where arrows pointing in opposite directions meet is where the evaluation metric is calculated.

4

Number of times uncertainty information caused model performance to be better, significantly better, the same and worse, than the equivalent model that did not use this information on the PubChem data sets. PB-RF, random forest with parametric bootstrap; W-RF, weighted random forest; VOS, random forest with variable output smearing; SVR, WSVR, weighted support vector regression.

5

Number of times uncertainty information caused model performance to be better, significantly better, the same and worse, than the equivalent model that did not use this information on the BASF data sets. PB-RF, random forest with parametric bootstrap; W-RF, weighted random forest; VOS, random forest with variable output smearing; SVR, WSVR, weighted support vector regression.

6

Change in root mean squared error as α is changed on data set AID 449756.