False discovery rates: a new deal

Abstract

We introduce a new Empirical Bayes approach for large-scale hypothesis testing, including estimating false discovery rates (FDRs), and effect sizes. This approach has two key differences from existing approaches to FDR analysis. First, it assumes that the distribution of the actual (unobserved) effects is unimodal, with a mode at 0. This "unimodal assumption" (UA), although natural in many contexts, is not usually incorporated into standard FDR analysis, and we demonstrate how incorporating it brings many benefits. Specifically, the UA facilitates efficient and robust computation-estimating the unimodal distribution involves solving a simple convex optimization problem-and enables more accurate inferences provided that it holds. Second, the method takes as its input two numbers for each test (an effect size estimate and corresponding standard error), rather than the one number usually used ($p$ value or $z$ score). When available, using two numbers instead of one helps account for variation in measurement precision across tests. It also facilitates estimation of effects, and unlike standard FDR methods, our approach provides interval estimates (credible regions) for each effect in addition to measures of significance. To provide a bridge between interval estimates and significance measures, we introduce the term "local false sign rate" to refer to the probability of getting the sign of an effect wrong and argue that it is a superior measure of significance than the local FDR because it is both more generally applicable and can be more robustly estimated. Our methods are implemented in an R package ashr available from http://github.com/stephens999/ashr.

Keywords: Empirical Bayes; False discovery rates; Multiple testing; Shrinkage; Unimodal.

Fig. 1.

Illustration that the UA in ash can produce very different results from existing methods. The figure shows, for a single simulated dataset, the way different methods decompose values (left) and scores (right) into a null component (dark blue) and an alternative component (cyan). In the score space the alternative distribution is placed on the bottom to highlight the differences in its shape among methods. The three existing methods (qvalue, locfdr, mixfdr) all produce a “hole” in the alternative score distribution around 0. In contrast ash makes the UA—that the effect sizes, and thus the scores, have a unimodal distribution about 0—which yields a very different decomposition. (In this case the ash decomposition is closer to the truth: the data were simulated under a model where all of the effects are non-zero, so the “true” decomposition would make everything cyan.)

Fig. 2.

Results of simulation studies (constant precision ). (a) Densities of non-zero effects, , used in simulations. (b) Comparison of true and estimated values of . When the UA holds all methods typically yield conservative (over-)estimates for , with ash being least conservative, and hence most accurate. qvalue is sometimes anti-conservative when . When the UA does not hold (“bimodal” scenario) the ash estimates are slightly anti-conservative. (c) Comparison of true and estimated from ash (ash.n). Black line is and red line is . Estimates of are conservative when UA holds, due to conservative estimates of . (d) As in (c), but for instead of . Estimates of are consistently less conservative than when UA holds, and also less anti-conservative in bimodal scenario.

Fig. 3.

Comparison of estimated cdfs from ash and the NPMLE. Different ash methods perform similarly, so only ash.hu is shown for clarity. Each panel shows results for a single example data set, one for each scenario in Figure 2(a). The results illustrate how the UA made by ash regularizes the estimated cdfs compared with the NPMLE.

Fig. 4.

Simulations showing how, with existing methods, but not ash, poor-precision observations can contaminate signal from good-precision observations. (a) Density histograms of values for good-precision, poor-precision, and combined observations. The combined data show less signal than the good-precision data, due to the contamination effect of the poor-precision measurements. (b) Results of different methods applied to good-precision observations only (-axis) and combined data (-axis). Each point shows the “significance” ( values from qvalue; for locfdr; for ash) of a good-precision observation under the two different analyses. For existing methods including the poor-precision observations reduces significance of good-precision observations, whereas for ash the poor-precision observations have little effect (because they have a very flat likelihood). (c) The relationship between and -value is different for good-precision () and low-precision () measurements: ash assigns the low-precision measurements a higher , effectively downweighting them. (d) Trade-off between true positives () vs false positives () as the significance threshold ( or value) is varied. By downweighting the low-precision observations ash re-orders the significance of observations, producing more true positives at a given number of false positives. It is important to note that this behaviour of ash depends on choice of . See Section 3.2.1 for discussion.