Inside the mind of a medicinal chemist: the role of human bias in compound prioritization during drug discovery

Abstract

Medicinal chemists' "intuition" is critical for success in modern drug discovery. Early in the discovery process, chemists select a subset of compounds for further research, often from many viable candidates. These decisions determine the success of a discovery campaign, and ultimately what kind of drugs are developed and marketed to the public. Surprisingly little is known about the cognitive aspects of chemists' decision-making when they prioritize compounds. We investigate 1) how and to what extent chemists simplify the problem of identifying promising compounds, 2) whether chemists agree with each other about the criteria used for such decisions, and 3) how accurately chemists report the criteria they use for these decisions. Chemists were surveyed and asked to select chemical fragments that they would be willing to develop into a lead compound from a set of ~4,000 available fragments. Based on each chemist's selections, computational classifiers were built to model each chemist's selection strategy. Results suggest that chemists greatly simplified the problem, typically using only 1-2 of many possible parameters when making their selections. Although chemists tended to use the same parameters to select compounds, differing value preferences for these parameters led to an overall lack of consensus in compound selections. Moreover, what little agreement there was among the chemists was largely in what fragments were undesirable. Furthermore, chemists were often unaware of the parameters (such as compound size) which were statistically significant in their selections, and overestimated the number of parameters they employed. A critical evaluation of the problem space faced by medicinal chemists and cognitive models of categorization were especially useful in understanding the low consensus between chemists.

Figure 1. Predictive accuracy of Semi-Naïve Bayesian (SNB) and Random Forest (RF) classifiers trained on medicinal chemists’ selections.

The average ROCS score for a 4-fold cross validation of each classifier is reported. A: SNB classifier built with medicinal chemistry relevant descriptors (red) is compared to a benchmark Naïve-Bayesian classifier that uses extended connectivity fingerprints and physical chemical properties as descriptors (black). B: RF classifier built with medicinal chemistry relevant descriptors (blue) is compared to a benchmark RF classifier that uses extended connectivity fingerprints and physical chemical properties as descriptors (black).

Figure 2. The parameters extracted from the SNB (red) and RF (blue) classifiers are compared with parameters designated as important in chemists’ self-reports (grey).

The primary parameters for the classifiers are depicted as stars, and the secondary parameters are depicted as circles. The one-tailed Fisher exact probability test (p) is reported for each parameter (except chains and charge), indicating that the SNB and RF parameters show agreement with each other, while the self reported parameters are independent of either of the classifier’s parameters.

Figure 3. Examples of selection preferences based on simple physicochemical properties, and the corresponding SNB classifiers.

A: Histogram of number of atoms of fragments selected by chemist 3 as good (green) or bad (red) starting points for drug discovery campaigns. Frequencies are normalized by the total number of selected or unselected compounds, respectively. B: Bayesian score versus number of atoms for minimal Bayesian model build for chemist 3. A positive score indicates a favorable number of atoms, while a negative score indicates an unfavorable number of atoms. C: Histogram of molecular polar surface area of fragments selected by chemist 12 as good (green) or bad (red) starting points for drug discovery campaigns. Frequencies are normalized by the total number of selected or unselected compounds, respectively. D: Bayesian score versus molecular polar surface area bins for SNB classifier built for chemist 12.

Figure 4. The SNB classifier built using a descriptor subsumed by the functional group parameter is illustrated for chemist 1.

Keys that represent the presence (black) or absence (white) of chemical substructures are ordered from negative (bad) on the left to positive (good) values on the right (A). The worst and best substructure keys are zoomed in on (B). Specific chemical substructures (tertiary amine – blue, aromatic heteroatom – violet, hydroxyl – aqua, and carboxylic acid - orange) are highlighted for one of the worst keys and two of the best keys, and illustrative examples of fragments that would be described by these keys are depicted (C).

Figure 5. Ring topology SNB classifier comparison between chemists.

The most favorable and unfavorable keys for the RingBonds_AromaticBonds_RingAssemblies (RB_AB_RA ) descriptor model, which measures the number of ring bonds (RB), aromatic bonds (AB), and ring assemblies (RA) present in a compound, were examined. Representative scaffolds that correspond to these keys are depicted, and are clustered based on how chemists viewed them. The Bayes score for each models built on individual chemists for each key is reported in a heat map. The favorable keys receive a positive score, while unfavorable keys receive a negative score.

Figure 6. The selection characteristics of chemists with high estimated consensus.

The cultural consensus model was applied to a subset of fragments (311) with >75% agreement by chemists. The estimated consensus obtained by this method is plotted against the fraction of fragments passed by chemists for the entire survey. Each shape describes the primary SNB parameter used to reproduce chemists’ selections, and the color depicts the ROC score of naïve Bayesian classifiers built using ECFP4 as a descriptor for each chemist. A subset of high consensus chemists is above the dashed grey line.