mQC: A Heuristic Quality-Control Metric for High-Throughput Drug Combination Screening

Abstract

Quality control (QC) metrics are critical in high throughput screening (HTS) platforms to ensure reliability and confidence in assay data and downstream analyses. Most reported HTS QC metrics are designed for plate level or single well level analysis. With the advent of high throughput combination screening there is a need for QC metrics that quantify the quality of combination response matrices. We introduce a predictive, interpretable, matrix-level QC metric, mQC, based on a mix of data-derived and heuristic features. mQC accurately reproduces the expert assessment of combination response quality and correctly identifies unreliable response matrices that can lead to erroneous or misleading characterization of synergy. When combined with the plate-level QC metric, Z', mQC provides a more appropriate determination of the quality of a drug combination screen. Retrospective analysis on a number of completed combination screens further shows that mQC is able to identify problematic screens whereas plate-level QC was not able to. In conclusion, our data indicates that mQC is a reliable QC filter that can be used to identify problematic drug combinations matrices and prevent further analysis on erroneously active combinations as well as for troubleshooting failed screens. The R source code of mQC is available at http://matrix.ncats.nih.gov/mQC.

Figure 1. Comparison of expert opinions of matrix-level quality and plate-level QC (Z’).

(A) A comparison from all 133 response matrices in the survey. (B) A comparison by removing bad-quality plates with Z’ < 0.

Figure 2. Performance of mQC.

(A) Heatmap of survey results and average error rate for each response matrix. (B) The multiclass MCC at different test set proportion using the original dataset (red) and Y-randomized dataset (blue). (C,D) The recall and precision of each matrix-level QC label at different test set proportion. (E) The confidence of mQC prediction as a function of the standard deviation of the predicted probabilities across mQC labels.

Figure 3

(A) Schematic workflow summarizing the steps involved in selecting the 133 combinations for the training set used to construct the mQC model. (B) Examples of response matrices and their mQC. Here we show 11 activity landscape in 3D and their corresponding “Bad”, “Medium” or “Good” classification predicted by mQC. Each activity landscape is transformed from the response matrix (see Methods for details) and annotated with respective surface features. We expect zero response on DMSO treatment (negative control), and a maximum 100% response (adjusted by the positive control) in CellTiter-Glo screens or −100% response (adjusted by the positive control) in Caspase-Glo screens.

Figure 4. QC metric and synergy distribution.

(A) Distribution of normalized delta-Bliss (DBNorm) with different levels of systematic error . (B) Distribution of DBNorm with different fractions of random error (p.random) introduced in the model. (C–E) Distribution of DBNorm based on mQC, Z’ or QC Mott et al. (F–H) Distribution of gamma based on mQC, Z’ or QC Mott et al.

Figure 5. Proposed QC guideline for drug combination screening.

(A) Each combination screening is represented by two independent points: a red point (Z’ as X-axis value and percentage of “Good” matrices as Y-axis), and a green point (Z’ as X-axis and percentage of “Good” plus “Medium” quality matrices as Y-axis). The distribution associated with Z’, Good%, Good + Medium% are beside the scatter plot. The dashed lines indicate the best practice cutoff for Z’ and mQC levels given a screening. (B) The best practice workflow for quality control of a cHTS campaign.

Figure 6. Comparison of readout, size of matrix and cell with respect to 7 feature distributions.

(A) Readout (Caspase-Glo(CG) vs. CellTiter-Glo(CTG)). (B) Size of matrix (6 × 6 vs. 10 × 10). (C) Cell. * = contaminated cell line. The arrows indicate the major difference between groups which significantly affects the mQC assessment.

Figure 7. Drift effect identified by mQC.

Each plot represents an independent screen consisting of 14 plates. The point in the upper part of plot represents the median response of matrix-level negative control (DMSO) on ith column. The point in the upper part of plot represents the proportion of “Good” response matrices on ith column, according to mQC assessment.