The Goldilocks paradigm: comparing classical machine learning, large language models, and few-shot learning for drug discovery applications

Abstract

Recent advances in machine learning (ML) have led to newer model architectures including transformers (large language models, LLMs) showing state of the art results in text generation and image analysis as well as few-shot learning (FSLC) models which offer predictive power with extremely small datasets. These new architectures may offer promise, yet the 'no-free lunch' theorem suggests that no single model algorithm can outperform at all possible tasks. Here, we explore the capabilities of classical (SVR), FSLC, and transformer models (MolBART) over a range of dataset tasks and show a 'goldilocks zone' for each model type, in which dataset size and feature distribution (i.e. dataset "diversity") determines the optimal algorithm strategy. When datasets are small ( < 50 molecules), FSLC tend to outperform both classical ML and transformers. When datasets are small-to-medium sized (50-240 molecules) and diverse, transformers outperform both classical models and few-shot learning. Finally, when datasets are of larger and of sufficient size, classical models then perform the best, suggesting that the optimal model to choose likely depends on the dataset available, its size and diversity. These findings may help to answer the perennial question of which ML algorithm is to be used when faced with a new dataset.

Fig. 1. Correlation plots between MolBART R², SVR R², the molecular diversity of the training set, and the number of molecules in the training dataset.

First, each of these metrics was calculated for each individual-target dataset in the original 2401 set from ChEMBL. The correlation was then calculated for each metric.

Fig. 2. Comparison of MolBART and SVR model R² with dataset size and molecule diversity.

A R² vs. the number of molecules for each of the 2401 training datasets from ChEMBL. Each point is a single-target dataset. B R² vs. the diversity for each of the 2401 training datasets. Each point is a single-target dataset.

Fig. 3. Example true vs. predicted -log(M) values for a small and large dataset for MolBART and SVR.

Perfect predictions would appear along the central gray line.

Fig. 4. Sample CSFP plot curves of a non-diverse dataset (MAP Kinase MNK1) and a diverse dataset (Protein Kinase C).

These images illustrate how the graph changes as diversity decreases. A perfectly diverse dataset with all unique scaffolds would be a straight diagonal line, while a dataset comprised of only one scaffold would encapsulate the entire area of the plot.

Fig. 5. Correlation plots between MolBART R²–SVR R², molecular diversity per training set (diversity), and the number of molecules per training set.

Each dot corresponds to a different target dataset. Correlations were significant between each paired feature (p < 0.05). Each datapoint represents a single-target dataset.

Fig. 6. Developing a Test Set for MARK1 Inhibition.

A The MedChemExpress FDA-Approved and Pharmacopeial Drug Library (HY-L066) was screened for MARK1 inhibition using the Promega ADP-Glo Kinase Assay at a concentration of 385 µM. Compounds that exhibited >90% inhibition are shown in light blue. B The Z-factor for each of the nine plates used in the screen. C IC₅₀ value determination for five novel MARK1 inhibitors using Z’-LYTE assay. Non-linear regression analysis (3-parameters) was performed in GraphPad Prism. Error bars are standard deviation.

Fig. 7. t-SNE plots of the MACCS key fingerprints of the kinase datasets and the discovered MARK1 inhibitors.

A Chemical space overlap of the kinase datasets. B Chemical space overlap of the discovered MARK1 inhibitors vs. the MARK1 dataset and the remaining kinase datasets.