Comparing Multiple Machine Learning Algorithms and Metrics for Estrogen Receptor Binding Prediction

Abstract

Many chemicals that disrupt endocrine function have been linked to a variety of adverse biological outcomes. However, screening for endocrine disruption using in vitro or in vivo approaches is costly and time-consuming. Computational methods, e.g., quantitative structure-activity relationship models, have become more reliable due to bigger training sets, increased computing power, and advanced machine learning algorithms, such as multilayered artificial neural networks. Machine learning models can be used to predict compounds for endocrine disrupting capabilities, such as binding to the estrogen receptor (ER), and allow for prioritization and further testing. In this work, an exhaustive comparison of multiple machine learning algorithms, chemical spaces, and evaluation metrics for ER binding was performed on public data sets curated using in-house cheminformatics software (Assay Central). Chemical features utilized in modeling consisted of binary fingerprints (ECFP6, FCFP6, ToxPrint, or MACCS keys) and continuous molecular descriptors from RDKit. Each feature set was subjected to classic machine learning algorithms (Bernoulli Naive Bayes, AdaBoost Decision Tree, Random Forest, Support Vector Machine) and Deep Neural Networks (DNN). Models were evaluated using a variety of metrics: recall, precision, F1-score, accuracy, area under the receiver operating characteristic curve, Cohen's Kappa, and Matthews correlation coefficient. For predicting compounds within the training set, DNN has an accuracy higher than that of other methods; however, in 5-fold cross validation and external test set predictions, DNN and most classic machine learning models perform similarly regardless of the data set or molecular descriptors used. We have also used the rank normalized scores as a performance-criteria for each machine learning method, and Random Forest performed best on the validation set when ranked by metric or by data sets. These results suggest classic machine learning algorithms may be sufficient to develop high quality predictive models of ER activity.

Keywords: Bayesian; deep learning; estrogen receptor; machine learning; support vector machine.

Figure 1.

Five-fold cross validation statistics for all eight datasets in this study organized by datasets (rows) and descriptors (columns). The green bar represents the best performing algorithm, measured by ACC, for a particular dataset-chemical space pair. All metrics were range scaled to [0, 1]. AdaBoost (ADA); Bernoulli Naïve-Bayes (BNB); Random Forest (RF); support vector classification (SVC); Deep Neural Networks (DNN).

Figure 2.

Validation set statistics for all eight datasets in this study. AdaBoost (ADA); Bernoulli Naive Bayes (BNB); Random forest (RF); support vector classification (SVC); Deep Neural Networks (DNN).

Figure 3.

Comparison of Assay Central model with other machine learning algorithms using a rank normalized score for the evaluation set. Assay Central (AC); AdaBoost (ADA); Bernoulli Naïve-Bayes (BNB); Random Forest (RF); support vector classification (SVC); Deep Neural Networks (DNN).