Consensus Modeling Strategies for Predicting Transthyretin Binding Affinity from Tox24 Challenge Data

Abstract

Transthyretin (TTR) is a key transporter of the thyroid hormone thyroxine, and chemicals that bind to TTR, displacing the hormone, can disrupt the endocrine system, even at low concentrations. This study evaluates computational modeling strategies developed during the Tox24 Challenge, using a data set of 1512 compounds tested for TTR binding affinity. Individual models from nine top-performing teams were analyzed for performance and uncertainty using regression metrics and applicability domains (AD). Consensus models were developed by averaging predictions across these models, with and without consideration of their ADs. While applying AD constraints in individual models generally improved external prediction accuracy (at the expense of reduced chemical space coverage), it had limited additional benefit for consensus models. Results showed that consensus models outperformed individual models, achieving a root-mean-square error (RMSE) of 19.8% on the test set, compared to an average RMSE of 20.9% for the nine individual models. Outliers consistently identified in several of these models indicate potential experimental artifacts and/or activity cliffs, requiring further investigation. Substructure importance analysis revealed that models prioritized different chemical features, and consensus averaging harmonized these divergent perspectives. These findings highlight the value of consensus modeling in improving predictive performance and addressing model limitations. Future work should focus on expanding chemical space coverage and refining experimental data sets to support public health protection.

1

Distribution plot of the experimental TTR binding activity for the training set (in green), leaderboard set (in blue), and blind test set (in red).

2

Empirical cumulative distribution functions (ECDF) of Tanimoto similarities for both leaderboard (blue) and blind test (red) sets. Plot A displays the maximum Tanimoto similarities, while plot B shows the mean Tanimoto similarities.

3

RMSE of predictions as a function of TTR binding activity (%) for each team model and the consensus model (Consensus I).

4

Scatter plots showing (A) Blind test and CV RMSE (%) of the models combined in the consensus (Table ). (B) Blind test and Leaderboard RMSE (%) as submitted to the challenge before the release of the leaderboard set. Each point represents a team, color-coded consistently across both plots. Dashed trend lines indicate Pearson correlation coefficients.

5

Williams plots for the five consensus models (from teams #2, #3, #6, #8, and #9), showing the relationship between absolute prediction errors and Consensus-STD values. Each plot includes both training (light blue ‘+’) and test (light green ‘x’) data points, with outliers marked in darker colors. The red lines represent bin-averaged RMSE values, establishing expected error thresholds for different Consensus-STD ranges. The vertical line indicates the AD threshold (at 90% of the training set).