Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Apr 8;64(7):2331-2344.
doi: 10.1021/acs.jcim.3c00799. Epub 2023 Aug 29.

MELLODDY: Cross-pharma Federated Learning at Unprecedented Scale Unlocks Benefits in QSAR without Compromising Proprietary Information

Wouter Heyndrickx  1 , Lewis Mervin  2 , Tobias Morawietz  3 , Noé Sturm  4 , Lukas Friedrich  5 , Adam Zalewski  6 , Anastasia Pentina  7 , Lina Humbeck  8 , Martijn Oldenhof  9 , Ritsuya Niwayama  10 , Peter Schmidtke  11 , Nikolas Fechner  4 , Jaak Simm  9 , Adam Arany  9 , Nicolas Drizard  12 , Rama Jabal  12 , Arina Afanasyeva  13 , Regis Loeb  9 , Shlok Verma  14 , Simon Harnqvist  14 , Matthew Holmes  14 , Balazs Pejo  15 , Maria Telenczuk  16 , Nicholas Holway  4 , Arne Dieckmann  17 , Nicola Rieke  18 , Friederike Zumsande  6 , Djork-Arné Clevert  7 , Michael Krug  5 , Christopher Luscombe  14 , Darren Green  14 , Peter Ertl  4 , Peter Antal  19 , David Marcus  14 , Nicolas Do Huu  12 , Hideyoshi Fuji  13 , Stephen Pickett  14 , Gergely Acs  15 , Eric Boniface  20 , Bernd Beck  8 , Yax Sun  21 , Arnaud Gohier  10 , Friedrich Rippmann  5 , Ola Engkvist  22 , Andreas H Göller  3 , Yves Moreau  9 , Mathieu N Galtier  23 , Ansgar Schuffenhauer  4 , Hugo Ceulemans  1
Affiliations

MELLODDY: Cross-pharma Federated Learning at Unprecedented Scale Unlocks Benefits in QSAR without Compromising Proprietary Information

Wouter Heyndrickx et al. J Chem Inf Model. .

Abstract

Federated multipartner machine learning has been touted as an appealing and efficient method to increase the effective training data volume and thereby the predictivity of models, particularly when the generation of training data is resource-intensive. In the landmark MELLODDY project, indeed, each of ten pharmaceutical companies realized aggregated improvements on its own classification or regression models through federated learning. To this end, they leveraged a novel implementation extending multitask learning across partners, on a platform audited for privacy and security. The experiments involved an unprecedented cross-pharma data set of 2.6+ billion confidential experimental activity data points, documenting 21+ million physical small molecules and 40+ thousand assays in on-target and secondary pharmacodynamics and pharmacokinetics. Appropriate complementary metrics were developed to evaluate the predictive performance in the federated setting. In addition to predictive performance increases in labeled space, the results point toward an extended applicability domain in federated learning. Increases in collective training data volume, including by means of auxiliary data resulting from single concentration high-throughput and imaging assays, continued to boost predictive performance, albeit with a saturating return. Markedly higher improvements were observed for the pharmacokinetics and safety panel assay-based task subsets.

PubMed Disclaimer

Conflict of interest statement

The authors declare the following competing financial interest(s): M.N.G. and M.T. are employed and own stocks in the company Owkin commercializing the underlying Federated Learning Platform based on the open source Substra software. The remaining authors have no conflicts of interest to declare.

Figures

Figure 1
Figure 1
Conceptual representation of the federated setup with two partners of different size, illustrating cross-end point and cross-compound federation. In practice, the number of end points amenable to cross-compound federation is far lower than to cross-end point federation, due to challenges in reconciliation across partners. Identical structures at different partners get identically represented, allowing implicit mapping through the machine learning algorithm without exchanging any sensitive information. Permission has been obtained to use the MELLODDY logo.
Figure 2
Figure 2
Overview of the different training modalities with layer sizes commonly optimal for partners for the federated setting (see SI for extensive optimal hyperparameters).
Figure 3
Figure 3
Performance deltas (between multi- and single-partner runs) based on median (top) and 90th percentile (bottom) across companies for their respective optimal model (either with or without auxiliary data).
Figure 4
Figure 4
Classification performance results from the federated run. (A) Effect of multipartner (MP) and auxiliary data (*) on the median AUC-PR task performance for 5 smaller (dashed lines) and 5 larger (solid lines) partners. (B) Distribution of median AUC-PR task performance (RIPtoP(AUC-PR)) over partners. (C–F) Difference between the empirical cumulative distribution functions (CDFs) from single- and multipartner models for different assay types based on AUC-ROC. The difference between the cumulative proportion of tasks in the multi- versus single-partner models (y-axis) is shown for the binned performance (x-axis). The line plots indicate the median probability difference for that bin over all partners. The interquartile ranges are indicated by the shaded envelope. Mind that AUC-ROC is shown here due to its stable baseline of 0.5 for a random classifier.
Figure 5
Figure 5
Classification applicability domain results from the federated run. (A) Effect of multipartner (MP) and auxiliary data (*) on the median task performance for 5 smaller (dashed lines) and 5 larger (solid lines) partners. (B) Distribution of median task performance (RIPtoP(CE)) over partners. (C–F) Difference between the empirical cumulative distribution functions (CDFs) from single- and multipartner models for different assay types based on CE. The difference between the cumulative proportion of tasks in the multi- versus single-partner models (y-axis) is shown for the binned performance (x-axis). The line plots indicate the median probability difference for a bin over partners. The interquartile ranges are indicated with the shaded envelope.
Figure 6
Figure 6
Regression performance results from the federated run. (A) Effect of multipartner (MP) and auxiliary data (*) on the median task performance for 5 smaller (dashed lines) and 5 larger (solid lines) partners. (B) Distribution of median task performance RIPtoP(R2)) over partners. (C–F) Difference between the empirical cumulative distribution functions (CDFs) from single- and multipartner models for different assay types based on R2. The difference between the cumulative proportion of tasks in the multi- versus single-partner models (y-axis) is shown for the binned performance (x-axis). The line plots indicate the median probability difference for a bin over partners. The interquartile ranges are indicated with the shaded envelope.
See this image and copyright information in PMC

References

    1. Hansch C.; Maloney P. P.; Fujita T.; Muir R. M. Correlation of Biological Activity of Phenoxyacetic Acids with Hammett Substituent Constants and Partition Coefficients. Nature 1962, 194, 178–180. 10.1038/194178b0. - DOI
    1. Hansch C.; Fujita T. ρ-σ-π Analysis. A Method for the Correlation of Biological Activity and Chemical Structure. J. Am. Chem. Soc. 1964, 86, 1616–1626. 10.1021/ja01062a035. - DOI
    1. Muratov E. N.; Bajorath J.; Sheridan R. P.; Tetko I. V.; Filimonov D.; Poroikov V.; Oprea T. I.; Baskin I. I.; Varnek A.; Roitberg A.; Isayev O.; Curtalolo S.; Fourches D.; Cohen Y.; Aspuru-Guzik A.; Winkler D. A.; Agrafiotis D.; Cherkasov A.; Tropsha A. QSAR without Borders. Chem. Soc. Rev. 2020, 49, 3525–3564. 10.1039/D0CS00098A. - DOI - PMC - PubMed
    1. Tang Y.; Chen K. X.; Jiang H. L.; Ji R. Y. QSAR/QSTR of Fluoroquinolones: An Example of Simultaneous Analysis of Multiple Biological Activities Using Neural Network Method. Eur. J. Med. Chem. 1998, 33, 647–658. 10.1016/S0223-5234(98)80023-8. - DOI
    1. González-Díaz H.; Prado-Prado F. J.; Santana L.; Uriarte E. Unify QSAR Approach to Antimicrobials. Part 1: Predicting Antifungal Activity against Different Species. Bioorg. Med. Chem. 2006, 14, 5973–5980. 10.1016/j.bmc.2006.05.018. - DOI - PubMed

Publication types

MeSH terms