Federated Learning in Computational Toxicology: An Industrial Perspective on the Effiris Hackathon

Abstract

In silico approaches have acquired a towering role in pharmaceutical research and development, allowing laboratories all around the world to design, create, and optimize novel molecular entities with unprecedented efficiency. From a toxicological perspective, computational methods have guided the choices of medicinal chemists toward compounds displaying improved safety profiles. Even if the recent advances in the field are significant, many challenges remain active in the on-target and off-target prediction fields. Machine learning methods have shown their ability to identify molecules with safety concerns. However, they strongly depend on the abundance and diversity of data used for their training. Sharing such information among pharmaceutical companies remains extremely limited due to confidentiality reasons, but in this scenario, a recent concept named "federated learning" can help overcome such concerns. Within this framework, it is possible for companies to contribute to the training of common machine learning algorithms, using, but not sharing, their proprietary data. Very recently, Lhasa Limited organized a hackathon involving several industrial partners in order to assess the performance of their federated learning platform, called "Effiris". In this paper, we share our experience as Roche in participating in such an event, evaluating the performance of the federated algorithms and comparing them with those coming from our in-house-only machine learning models. Our aim is to highlight the advantages of federated learning and its intrinsic limitations and also suggest some points for potential improvements in the method.

Figure 1

Representation of the main Effiris federated learning scheme for each of the company members.

Figure 2

Representation of the Effiris SOHN federated learning scheme. The internal member data are used to train the “teacher” model (A), while both internal and Lhasa public data are combined to train the “hybrid” model (B). The final “hybrid federated” model is trained on the combination of internal, public, and federated data (C).

Figure 3

Representation of the Effiris MLP federated learning scheme. (A) An initial model is trained on public data curated by Lhasa. This pretrained model is then refined with internal data in order to get the “fine-tuned” MLP. (B) The federated labels obtained from predictions of consolidated “fine-tuned models” on the FLuID data from the members are then used to train another model, called “pretrained federated”, which is further refined with member data, generating the “fine-tuned federated” model.

Figure 4

t-SNE plots representing the distribution of the chemical space for the AChM₁R data sets. Specifically, panel A depicts the reduction of the pool of features composed by the 208 RDKit physicochemical descriptors and the 2048-bits fingerprints to just two dimensions, while in panel B, the three-dimensional reduction is reported.

Figure 5

Mathematical formula for the Matthews Correlation Coefficient (MCC) calculation. The acronyms are the following: TP = True Positive; TN = True Negative; FP = False Positive; FN = False Negative).

Figure 6

Trends in the Applicability Domain (AD) compared to the main performance metrics considered in the present study of the Lhasa SOHN architecture. The AD versus the sensitivity, specificity, balanced accuracy, and MCC of the prediction on the external test set are reported in (A–D), respectively. As the legend shows, the purple line indicates the data coming from the predictions on the COX2 data set, the green one stands for GABA_A, the cyan for the hERG channel, the dashed blue line for AChM₁R, and the dashed black line depicts the outcomes for 5-HT_2B. The different data sets considered are also indicated with a different letter, as reported below. Specifically, T indicates metrics and AD for the “teacher” SOHN model (trained just on the internal Roche data); HI stands for “hybrid internal” and represents the SOHN model trained on both Lhasa public data and Roche internal data. Finally, the acronym “HF”, or “hybrid federated”, indicates the SOHN model trained on both federated data and Roche internal data.

Figure 7

Trends in the Applicability Domain (AD) compared to the main performance metrics considered in the studies presented here for the Lhasa MLP architecture. Specifically, the AD versus the sensitivity, specificity, balanced accuracy, and MCC of the prediction on the external test set are reported in (A–D), respectively. As the legend represents, the purple line indicates the data coming from the predictions on the COX2 data set, the green one stands for GABA_A, the cyan for the hERG channel, the dashed blue line for AChM₁R, and the dashed black line depicts the outcomes for 5-HT_2B. The different data sets considered are also indicated with a different letter, as reported below. Specifically, PI indicates metrics and AD for the pretrained internal MLP model (trained just on the Lhasa public data); RI stands for “refined internal” and represents the MLP model trained on Lhasa Limited public data and then refined with Roche internal data. Then, the letter “F”, or “federated”, indicates the MLP model trained on just the federated data, while “RF” (“refined federated”) represents the MLP trained on the federated data and then refined using Roche internal data.

Figure 8

Plots representing the metrics of the predictions on the external test set, using the different training setups discussed in the previous paragraph (for both internal and Lhasa models). A–D represent the sensitivity, specificity, balanced accuracy, and MCC metrics for each of the cases considered, respectively. For the first three plots, also the 0.5 value is marked with a dashed red line, indicating the point below which the predictions start to lose relevance (half of the positives/negatives are misclassified by the algorithm).