A versatile information retrieval framework for evaluating profile strength and similarity

Abstract

Large-scale profiling assays capture a cell population's state by measuring thousands of biological properties per cell or sample. However, evaluating profile strength and similarity remains challenging due to the high dimensionality and non-linear, heterogeneous nature of measurements. Here, we develop a statistical framework using mean average precision (mAP) as a single, data-driven metric to address this challenge. We validate the mAP framework against established metrics through simulations and real-world data, revealing its ability to capture subtle and meaningful biological differences in cell state. Specifically, we use mAP to assess a sample's phenotypic activity relative to controls, as well as the phenotypic consistency of groups of perturbations (or samples). We evaluate the framework across diverse datasets and on different profile types (image, protein, mRNA), perturbations (CRISPR, gene overexpression, small molecules), and resolutions (single-cell, bulk). The mAP framework, together with our open-source software package copairs, is useful for evaluating high-dimensional profiling data in biological research and drug discovery.

Fig. 1. Schematic overview of the mAP framework.

A A typical output of a profiling experiment contains multiple replicate profiles for each perturbation and controls. B To measure average precision (AP) per perturbation replicate, we selected one replicate profile as a query and measured distances to its other replicates and controls. C Profiles were then ranked by decreasing similarity (increasing distance) to the query; the rank list was converted to binary form and used to calculate precision P_k and recall R_k at each rank k. D Average precision was calculated by averaging precision values over those ranks k containing perturbation replicates, which corresponds to a non-interpolated approximation of the area under the precision-recall curve. E By applying this procedure to each perturbation replicate, we calculated a set of AP scores that were then averaged to obtain a mAP score for a perturbation’s phenotypic activity. F One can also apply the same framework to retrieving groups of perturbations with the same biological annotations (rather than groups of replicates of the same perturbation)—for example, compounds that share the same mechanism of action (MoA)—by calculating the mAP score per each group of perturbations (MoA). Source data are provided as a Source Data file.

Fig. 2. The mAP framework evaluation on simulated data.

Benchmarking retrieval performance of mAP p-value (orange), mp-value (blue), MMD p-value (green), and k-means clustering (purple) for retrieving phenotypic activity on simulated data, where unperturbed and perturbed features are sampled from $\mathcal{N}$ (0,1) and $\mathcal{N}$ (1,1), correspondingly. Recall indicates the percentage of 100 simulated perturbations under each condition that were called accurately by each method (as distinguishable from negative controls, or not). The horizontal axis probes what proportion of the features in the profile were different from controls (note the binary exponential scaling). Marker and line styles indicate different numbers of replicates per perturbation (# replicates of 2, 3, and 4). Columns correspond to the different number of controls (# controls of 12, 24, and 36). Rows correspond to different profile sizes (# features being 100, 200, 500, 1000, 2500, and 5000). mAP, mp-value, and MMD used a one-sided permutation test to obtain p-values without adjusting for multiple comparisons; no statistical test was performed for k-means. Source data are provided as a Source Data file.

Fig. 3. The mAP framework applied to morphological profiling of CRISPR-Cas9 knockout perturbations (Cell Health dataset).

A mAP is calculated to assess well position and individual plate effects on phenotypic activity by retrieving guide replicates against controls in three scenarios (replicates of a guide across different plates and well positions; replicates of a guide across different plates, but in the same well position; and replicates of a guide within the same plate, but in different well position) and two data preprocessing methods (standardize and MAD robustize per plate). Percentages retrieved indicate the percentage of scores with p-value below 0.05 per cell line (and averaged across all cell lines in parenthesis). B mAP is calculated to assess the phenotypic activity of perturbations by guide replicate retrievability against controls in three cell lines individually (49% retrieved on average across all three cell lines). Results included all three replicate plates available per cell line. C Replicate-level AP scores calculated for a subset of guides from (B) highlight the variation from guide to guide across cell lines. D mAP p-values estimated to assess the influence of individual fluorescence channels on guide phenotypic activity against controls by either dropping a channel or including only that single channel (percent retrieved is shown for each axis); these results can be compared to 49% retrieved when all channels’ data is available (on average across all three cell lines, as in B). E mAP is calculated to assess the phenotypic consistency of guides annotated with related target genes (against guides annotated with other genes) in three cell lines individually. F Guide-level AP scores calculated for a subset of genes from (E) highlight the variation from gene to gene  across cell lines. mAP p-values were estimated using a one-sided permutation test and adjusted for multiple comparisons by Benjamini–Hochberg procedure. Percent retrieved indicates the percentage of scores with p-value below 0.05. Source data are provided as a Source Data file.

Fig. 4. The mAP framework applied to proteomic and mRNA profiling.

A mAP is calculated to assess the phenotypic activity of compounds by replicate retrievability against controls in matching Cell Painting and nELISA profiling data. B A combined view of the data from (A) is presented, showing phenotypic activity retrieval for both assays. C mAP is calculated to assess the phenotypic consistency by retrieving phenotypically active compounds annotated with the same gene target in matching Cell Painting and nELISA profiling data (note: the nELISA panel includes 191 targets including cytokines, chemokines, and growth factors which are not expected to respond well in these convenience samples from a prior study, because there is no immune stimulation and the A549 cells used have limited secretory capacity). D A combined view of the data from (C) is presented, showing phenotypic consistency retrieval for both assays. E mAP is calculated to assess the mRNA profile-based phenotypic activity of a mismatched CRISPRi guide from a Perturb-seq experiment (y-axis) and correlate it with the guide’s activity relative to a perfectly matching guide for that gene (x-axis). A linear model fit is shown in black with gray error bands showing the 95% confidence interval. F A subset of the data from (E) is presented, with several genes highlighted individually to demonstrate the variation from gene to gene. mAP p-values were estimated using a one-sided permutation test and adjusted for multiple comparisons by Benjamini–Hochberg procedure. Percent retrieved indicates the percentage of scores with p-value below 0.05. Source data are provided as a Source Data file.

Fig. 5. The mAP framework applied to single-cell mRNA and imaging data.

A AP scores are calculated to assess the single-cell mRNA profile-based phenotypic activity of a mismatched CRISPRi guide from a Perturb-seq experiment (y-axis) and correlate it with the guide’s activity relative to a perfectly matching guide for that gene (x-axis). B A subset of the data from (A) is presented, with several genes highlighted individually to demonstrate the variation from gene to gene. C AP scores are calculated to evaluate the power of CellProfiler and DeepProfiler features to classify multiple phenotypic classes in Mitocheck morphological data. AP scores capture the ability to retrieve single cells annotated with the same morphological class against negative controls. D Mitocheck data, correlation between mAP scores for retrieving single cells annotated with the same morphological class versus gene, for DeepProfiler features. MC: morphological class. mAP p-values were estimated using a one-sided permutation test and adjusted for multiple comparisons by Benjamini–Hochberg procedure. Percent retrieved indicates the percentage of scores with p-value below 0.05. Source data are provided as a Source Data file.