Human-in-the-loop assisted de novo molecular design

Abstract

A de novo molecular design workflow can be used together with technologies such as reinforcement learning to navigate the chemical space. A bottleneck in the workflow that remains to be solved is how to integrate human feedback in the exploration of the chemical space to optimize molecules. A human drug designer still needs to design the goal, expressed as a scoring function for the molecules that captures the designer's implicit knowledge about the optimization task. Little support for this task exists and, consequently, a chemist usually resorts to iteratively building the objective function of multi-parameter optimization (MPO) in de novo design. We propose a principled approach to use human-in-the-loop machine learning to help the chemist to adapt the MPO scoring function to better match their goal. An advantage is that the method can learn the scoring function directly from the user's feedback while they browse the output of the molecule generator, instead of the current manual tuning of the scoring function with trial and error. The proposed method uses a probabilistic model that captures the user's idea and uncertainty about the scoring function, and it uses active learning to interact with the user. We present two case studies for this: In the first use-case, the parameters of an MPO are learned, and in the second use-case a non-parametric component of the scoring function to capture human domain knowledge is developed. The results show the effectiveness of the methods in two simulated example cases with an oracle, achieving significant improvement in less than 200 feedback queries, for the goals of a high QED score and identifying potent molecules for the DRD2 receptor, respectively. We further demonstrate the performance gains with a medicinal chemist interacting with the system.

Keywords: AI-assisted design; De novo molecular design; Expert knowledge elicitation; Goal-oriented molecule generation; Human-in-the-loop; Interactive algorithms; Reward elicitation.

Fig. 1

Human-in-the-loop de novo molecular design: an AI-assistant helps a chemist to decide parameters of an MPO objective function $S_{r,t}\left(x\right)$ iteratively at round $r$ and iteration $t$, where $r$ are rounds of goal-directed molecule generation with a de novo design tool, and $t$ are online interactions with a chemist. The objective consists of K molecular properties $c_k\left(x\right)$ with relative weights $w_k$. The utility of the $k$:th property is measured using a desirability function ${\varphi }_{r,t,k}$ that defines the range of good property values. At each iteration, the method selects a molecule $x_{r,t}$ to query, which the chemist evaluates with feedback $y$. The method then adapts $S_{r,t}\left(x\right)$ based on the feedback by estimating the parameters of ${\varphi }_{r,t,k}$ to match the chemist’s underlying goal

Fig. 2

Graphical user interface for giving feedback to molecules. The chemist evaluates DRD2 activity of molecules on a scale from 1 to 5 For initialization, we randomly sample 10 molecules and get their scores from the oracle. For the experiment with a human chemist, we randomly sample 10,000 molecules to be unlabeled molecules $\mathcal{U}$ to speed up the method. For ten iterations we sequentially query 100 molecules in batches of 10 from a chemist, who evaluates them on a scale from 1 to 5 (0 = very likely not active, 5 = very likely active). The scores are linearly scaled to the range [0,1]. The order of the evaluated molecules is chosen using Thompson sampling that was the best in the simulated experiments. For evaluating the performance, the oracle model is used to score the molecules generated by REINVENT with the chemist’s component as a scoring function at iteration $t=1,\cdots ,10$.

Fig. 3

The parameters of the MPO objective are better estimated with increasing amount of feedback. The mean relative absolute error (MRAE) in the estimated parameters decreases with increasing human feedback, and fastest with Thompson sampling. Solid lines show average of MRAE over 10 random seeds, and the shaded areas one standard error of the mean (SEM)

Fig. 4

The average oracle score of the generated molecules increases at each round of adapting the MPO. At each round, a new batch of molecules is generated using an adapted scoring function after in total 110 queries (round 1) and 220 queries (round 2) to a simulated chemist. For comparison, we show round 0 that is the performance with the initial guess ${\theta }_0$. The bars show the mean of the average oracle score of the generated molecules over 10 random seeds, and the error bars represent one SEM. The gray horizontal line shows the average oracle score in 5000 molecules sampled from REINVENT without MPO objective, using its prior agent

Fig. 5

A non-parametric scoring component that represents the chemist’s knowledge improves REINVENT output even with small number of queries (< 100) to a simulated chemist. The lines show the average oracle score in the REINVENT output and shaded areas its variation in 10 repeated experiments (mean and SEM). The method is not very sensitive to Gaussian noise in the simulated chemist’s answers. a noise level ${\sigma }_{\mathrm{chemist}}=0.0$, b ${\sigma }_{\mathrm{chemist}}=0.15$, c ${\sigma }_{\mathrm{chemist}}=0.30$

Fig. 6

A medicinal chemist’s feedback on DRD2 activity of molecules improves the average activity of the generated molecules, measured using activity prediction model described in “Task 2: Learn human knowledge about a molecular property as a separate component” section. The dashed lines show performance in three repeated experiments. The performance is summarized in mean performance (solid line) with one standard error of mean (shaded area). The repetitions differ by different randomly sampled initial data and consequently different actively selected queries. The queries are selected using Thompson sampling