Predicting Organ Toxicity Using in Vitro Bioactivity Data and Chemical Structure

Abstract

Animal testing alone cannot practically evaluate the health hazard posed by tens of thousands of environmental chemicals. Computational approaches making use of high-throughput experimental data may provide more efficient means to predict chemical toxicity. Here, we use a supervised machine learning strategy to systematically investigate the relative importance of study type, machine learning algorithm, and type of descriptor on predicting in vivo repeat-dose toxicity at the organ-level. A total of 985 compounds were represented using chemical structural descriptors, ToxPrint chemotype descriptors, and bioactivity descriptors from ToxCast in vitro high-throughput screening assays. Using ToxRefDB, a total of 35 target organ outcomes were identified that contained at least 100 chemicals (50 positive and 50 negative). Supervised machine learning was performed using Naïve Bayes, k-nearest neighbor, random forest, classification and regression trees, and support vector classification approaches. Model performance was assessed based on F1 scores using 5-fold cross-validation with balanced bootstrap replicates. Fixed effects modeling showed the variance in F1 scores was explained mostly by target organ outcome, followed by descriptor type, machine learning algorithm, and interactions between these three factors. A combination of bioactivity and chemical structure or chemotype descriptors were the most predictive. Model performance improved with more chemicals (up to a maximum of 24%), and these gains were correlated (ρ = 0.92) with the number of chemicals. Overall, the results demonstrate that a combination of bioactivity and chemical descriptors can accurately predict a range of target organ toxicity outcomes in repeat-dose studies, but specific experimental and methodologic improvements may increase predictivity.

Figure 1.

Distribution of positive and negative chemicals across the in vivo guideline toxicity testing studies and target organs. From left to right these bar graphs show the number of positive (pos, red) and negative (neg, green) chemicals for chronic (CHR), subchronic (SUB), multigenerational (MGR) and developmental (DEV) studies. The target organs are labeled on the ordinate and the number of chemicals on the abscissa. The negative chemicals are missing for guideline studies where the evaluation of the specific target organ effect was not compulsory.

Figure 2.

Relationship between F1 score and number of descriptors for the best performing classification models and illustrative examples of minimal datasets. In each graph, the effect and descriptor type are given in the title (denoted as study:target-organ), the mean F1 score, and the standard deviation is shown in blue and gray, respectively. The number of descriptors and F1 score for the best classifier are signified on the ordinate and abscissa, respectively, by vertical and horizontal red lines. Each graph shows the cross-validation F1 score (ordinate) and number of descriptors (abscissa) for predicting toxicities (shown in the title and denoted as study:target-organ) using classification methods (shown in title)

Figure 3.

Summary of performance for target organ outcomes for select minimum datasets by classification algorithms and descriptors. The visualization shows the predictive performance for illustrative examples of target organ outcomes in rows (denoted as, study:target organ) using eight machine learning algorithms (columns): naïve Bayes (NB), k-nearest neighbor classification (KNN0 and KNN1) classification and regression trees (CART0 and CART1) and support vector classifiers (SVCL0 and SVCR0). The predictive performance is compared across five different descriptors including: chemical (chm), chemotype (ct), in vitro bioactivity (bio), a combination of in vitro bioactivity and chemical (bc), and a combination of in vitro bioactivity and chemotype (ct). The performance of a classification method for predicting an outcomeusing a descriptor type was measured using specificity (green), F1 score (red) and sensitivity (blue), which are visualized as vertical glyphs. The center, top, and bottom of the glyphs correspond to the mean ±1 SD. In all, the performance results for 40 classification methods (8 machine learning algorithms and five descriptor types) are visualized for each target organ toxicity. The grey horizontal bars on each graph signify the best mean F1 score ±1 SD (across all 5-fold cross-validation trials). The best performing classification model and descriptor set for each target organ outcome are denoted with a vertical red line.

Figure 4.

Summary of frequently used bioactivity descriptors in chronic target organ toxicity prediction models. The visualization shows a heat map in which the rows correspond to chronic target organ toxicities, columns correspond to the fifty most frequently used bioactivity descriptors, and values represent row standardized frequencies of occurrence of descriptors (column) in predictive models of target organ toxicities (row). The colors signify the row standardized frequencies for the bioactivity descriptors where positive values are red, negative values are blue and the level of saturation is directly related to magnitude. The row dendrogram show the cosine similarity between the frequency of bioactivity descriptors and target organ toxicity outcomes, respectively, by average linkage clustering.

Figure 5.

Summary of frequently used chemotype descriptors in chronic target organ toxicity prediction models. The visualization shows a heat map in which the rows correspond to chronic target organ toxicities, columns correspond to the fifty most frequently used chemotype descriptors, and values represent row standardized frequencies of occurrence of descriptors (column) in predictive models of target organ toxicities (row). The colors signify the row standardized frequencies for the chemotype descriptors where positive values are red, negative values are blue and the level of saturation is directly related to magnitude. The row dendrogram shows the cosine similarity between the frequency of chemotype descriptors and target organ toxicity outcomes, respectively, by average linkage clustering.