The ENCODE Imputation Challenge: a critical assessment of methods for cross-cell type imputation of epigenomic profiles

Abstract

A promising alternative to comprehensively performing genomics experiments is to, instead, perform a subset of experiments and use computational methods to impute the remainder. However, identifying the best imputation methods and what measures meaningfully evaluate performance are open questions. We address these questions by comprehensively analyzing 23 methods from the ENCODE Imputation Challenge. We find that imputation evaluations are challenging and confounded by distributional shifts from differences in data collection and processing over time, the amount of available data, and redundancy among performance measures. Our analyses suggest simple steps for overcoming these issues and promising directions for more robust research.

Fig. 1

Results from the ENCODE Imputation Challenge. A The H3K27ac signal for brain microvascular endothelial cells that is observed (in blue), from baseline methods, and from the winning three teams in the challenge. B The same as A except for DNase-seq signal in DND-41 cells. C The average MSE for each method across test set tracks and bootstraps but partitioned by assay type. D The same as C except for Pearson correlation. E The overall score, calculated as described in the “Performance measures” section, across all test set tracks and performance measures shown for each bootstrap for each team. The baseline methods and winners are colored

Fig. 2

Distributional shift and quantile normalization. A Experimental signal measuring H3K4me3 in BE2C cells from an unnormalized training set experiment (gray), an unnormalized test set experiment in SJSA1 cells (green), the test set signal after quantile normalization (blue), the test set signal after single-end reprocessing (red), and the test set signal after single-end reprocessing and quantile normalization (purple). B Distributions of signal values within peaks in chr16/17 for each reprocessed assay across the unnormalized training set (gray), the unnormalized test set (green), the single-end reprocessed test set (red), and the single-end reprocessed and quantile-normalized test set (purple). The KS statistics between the training set distribution and the test set distributions are shown in the legends and the CDFs are summarized using 25 dots for visualization purposes. C An example locus that exhibits a DNase peak in both the training and test sets. D A re-scoring of the challenge participants against single-end reprocessed and quantile-normalized test set signal

Fig. 3

Additional performance measures. A Experimentally observed signal for H3K27ac in brain microvascular endothelial cells. B An example of partitioning the track from A into logarithmically spaced bins (the rows). C The accuracy between binarized imputations and MACS2 peak calls for each signal bin when using the experimental signal to define the bins. D The same as C except using the imputed signal to define the bins. E The same as A but a different locus. F The same as B except calculating bins using the number of cell types that each locus exhibits a peak in. G The precision of the binarized imputed signal against MACS2 peak calls when evaluated separately for each bin. H The same as G except the recall instead of the precision. I The average area under the curves, calculated as shown in C, across all test set tracks for each participant. J The average area under the curves calculated as shown in D across all test set tracks for each participant. K The precision score calculated in the same manner as I/J. L The same as K, except the recall score. M The average H3K4me3 profile of experimental (blue), quantile-normalized (magenta), and imputed signals at strand-corrected promoters. N The average Pearson correlation between imputed and quantile-normalized signal across all promoters and H3K4me3 test set tracks. O The average Pearson correlation between imputed and quantile-normalized signal across all observed DNase peaks and DNase test set tracks

Fig. 4

The challenge data matrix. The matrix shows the experiments used in the challenge, colored based on whether they were in the training set (blue), the validation set (orange), or the blind test set (green). White squares indicate that an experiment has not yet been performed. The marginal bar plots show the number of experiments in each assay and cell type