A Bayesian measure of the probability of false discovery in genetic epidemiology studies

Abstract

In light of the vast amounts of genomic data that are now being generated, we propose a new measure, the Bayesian false-discovery probability (BFDP), for assessing the noteworthiness of an observed association. BFDP shares the ease of calculation of the recently proposed false-positive report probability (FPRP) but uses more information, has a noteworthy threshold defined naturally in terms of the costs of false discovery and nondiscovery, and has a sound methodological foundation. In addition, in a multiple-testing situation, it is straightforward to estimate the expected numbers of false discoveries and false nondiscoveries. We provide an in-depth discussion of FPRP, including a comparison with the q value, and examine the empirical behavior of these measures, along with BFDP, via simulation. Finally, we use BFDP to assess the association between 131 single-nucleotide polymorphisms and lung cancer in a case-control study.

Figure 1.

Graphical representation of BFDP (a) and FPRP (b). For BFDP, the approximate Bayes factor is the ratio of indicated densities, whereas, for FPRP, the dark- and light-shaded areas represent and , respectively.

Figure 2.

Distribution of BFDP by allele frequency for 1,000 cases and 1,000 controls with a relative risk of 1.5 and with π₀=0.99. The solid line connects the median BFDP at each frequency. The dotted line corresponds to a noteworthy threshold of 0.8; the powers to achieve this level (i.e., for the BFDP to fall below this threshold) at the given frequencies are 0%, 13%, 42%, 76%, 88%, 92%, and 92%.

Figure 3.

Number of cases required to obtain a BFDP of <0.8 with probability 0.8, as a function of the frequency of one or two mutant allele copies and π₁, with a relative risk of 1.5 (under the assumption of an equal number of cases and controls).

Figure 4.

Log relative risk versus MAF for the 100 nonnull SNPs in the simulated data. Blackened and nonblackened circles represent SNPs called noteworthy and nonnoteworthy, respectively, by BFDP at the stated threshold.

Figure 5.

Operating characteristics of BFDP. Panels a, b, c, and d correspond to possibilities U, V, T, and S in table 2, respectively. In each panel, the solid lines represent the numbers of true nonnoteworthy, false nonnoteworthy, false nonnoteworthy, and true noteworthy tests at each threshold for BFDP. The dashed lines are the posterior expected numbers of tests that are true nonnoteworthy, false nonnoteworthy, false nonnoteworthy, and true noteworthy. The dotted lines represent the ideal outcome that a perfect test would achieve.

Figure 6.

Operating characteristics of FPRP. Panels a, b, c, and d correspond to possibilities U, V, T, and S in table 2, respectively. In each panel, the solid lines represent the numbers of true nonnoteworthy, false nonnoteworthy, false nonnoteworthy, and true noteworthy tests at each threshold for FPRP. The dashed lines are the posterior expected numbers of tests that are true nonnoteworthy, false nonnoteworthy, false nonnoteworthy, and true noteworthy. The dotted lines represent the ideal outcome that a perfect test would achieve.

Figure 7.

Comparison between BFDP, FPRP, P, and q for the 5,000 SNPs with the lowest values of BFDP. a, BFDP versus FPRP. b, P values versus FPRP. c, q values versus FPRP. d, q values versus BFDP.

Figure 8.

Boxplots of BFDP (a–c) and FPRP (d–f). π₀ values are shown above each panel. For panels a–c, each set of boxplots corresponds to θ_p=1.50, 1.88, 2.25, 2.63, and 3.00 (the 97.5% point of the prior); for panels d–f, we evaluate the power at θ₁=1.50, 1.88, 2.25, 2.63, and 3.00. The dashed lines in panels a–c denote the thresholds that correspond to false nondiscovery being three times as costly as false discovery, so that SNPs with BFDP values below the line are noteworthy.

Figure B1.

Approximate Bayes factor versus exact Bayes factor as a function of the number of controls, n₀; the number of cases, n₁; and the log relative risk, θ. The MAF is 0.05, and there are 50 simulated data sets in each plot. In panel a, n₀=n₁=250 and θ=log(1.0). In panel b, n₀=n₁=250 and θ=log(1.3). In panel c, n₀=n₁=500 and θ=log(1.0). In panel d, n₀=n₁=500 and θ=log(1.3).