Predictive modeling of biodegradation pathways using transformer architectures

Abstract

In recent years, the integration of machine learning techniques into chemical reaction product prediction has opened new avenues for understanding and predicting the behaviour of chemical substances. The necessity for such predictive methods stems from the growing regulatory and social awareness of the environmental consequences associated with the persistence and accumulation of chemical residues. Traditional biodegradation prediction methods rely on expert knowledge to perform predictions. However, creating this expert knowledge is becoming increasingly prohibitive due to the complexity and diversity of newer datasets, leaving existing methods unable to perform predictions on these datasets. We formulate the product prediction problem as a sequence-to-sequence generation task and take inspiration from natural language processing and other reaction prediction tasks. In doing so, we reduce the need for the expensive manual creation of expert-based rules.

Keywords: Biodegradation; Cheminformatics; Product prediction; Transfer-learning; Transformer.

Fig. 1

How pathway prediction is performed. A root molecule is given to a prediction method which recursively predicts products from reactants until an arbitrary stopping condition is reached

Fig. 2

An example reaction SMILES where 1,2,4-triazole degrades into Triazole-alanine

Fig. 3

An overview of the enviFormer method. During training reactants and products are extracted from the train-set reactions and fed to the transformer. The transformers output is then compared with the true product to calculate the loss for the model to learn from. During inference a reactant SMILES is given to the transformer, which generates a list of potential products with associated probabilities. Depending on the threshold used different products will form the final set of output products

Fig. 4

This example is different reactions 1,2,4-triazole undergoes taken from the Soil dataset. There are two cases of multiple products, separate reactions represented by two different SMILES and one reaction producing multiple products represented by one SMILES

Fig. 5

A beam decoding example using a simple ABC set of tokens to show how beam decoding is performed, starting with the SOS token. In this example, the beam width is two, and the two output sequences are ABC and CAC, with probabilities of 0.38 and 0.09, respectively

Fig. 6

Multi Generation evaluation example. Note that compound F does not contribute to the score as it is an intermediate product and compound D gets its score adjusted as if it was an immediate product of compound B. With a threshold of 0.3, we get $Precision=(B+D)/(B+D)=1$, and $Recall=(B+D)/(B+C+D)=0.75$

Fig. 7

We show enviFormer’s performance on different training sets. Comparing the cyan USPTO line to other shows the benefits of transfer learning

Fig. 8

A comparison to the models trained with rules extracted by experts and extracted by enviRule on the BBD dataset

Fig. 9

A comparison to the rules extracted by experts and extracted by enviRule on the soil dataset

Fig. 10

A comparison to the models trained with rules extracted by experts and extracted by enviRule from BBD and Soil and then evaluated on the Sludge dataset

Fig. 11

The runtime of enviFormer with (green) and without (orange) GPU acceleration compared to the hybrid rule based method (blue). Pathway represents the prediction time from the root. For the batch sizes the time reported is the time per batch. * We used a Python implementation of the Ensemble of Classifier Chains method utilised by enviPath [8] with 317 rules extracted by enviRule