Comments on the analysis of unbalanced microarray data

Abstract

Motivation: Permutation testing is very popular for analyzing microarray data to identify differentially expressed (DE) genes; estimating false discovery rates (FDRs) is a very popular way to address the inherent multiple testing problem. However, combining these approaches may be problematic when sample sizes are unequal.

Results: With unbalanced data, permutation tests may not be suitable because they do not test the hypothesis of interest. In addition, permutation tests can be biased. Using biased P-values to estimate the FDR can produce unacceptable bias in those estimates. Results also show that the approach of pooling permutation null distributions across genes can produce invalid P-values, since even non-DE genes can have different permutation null distributions. We encourage researchers to use statistics that have been shown to reliably discriminate DE genes, but caution that associated P-values may be either invalid, or a less-effective metric for discriminating DE genes.

Fig. 1.

QValue estimates of π₁ for the Δμ-statistic. A reference line is drawn at π₁^μ, the proportion of genes DE in the mean. However, π₁^μ=π₁ only for the EV simulations. For the UV simulations, π₁^μ<π₁. CDE, an alternative MMM, gives similar results (Supplementary Fig. 3). See also Supplementary Figures 4 and 5 for the s-statistic.

Fig. 2.

ROC curves for the Δμ-statistic (dashed line), s-statistic (solid line) and t-statistic (dotted line) in a type EV simulation with 10 000 genes. The ROC curves for the corresponding permutation test P-values are the same as for the t-statistic and are not shown.

Fig. 3.

Histograms of ‘per gene’ permutation null distributions for nine genes in UV simulations. All nine genes have the same mean in both populations, samples sizes are 60 and 45, and histograms are based on 2000 random permutations. There is clearly gene-to-gene variation in the permutation null distributions of these test statistics.

Fig. 4.

A comparison of the P-values from a ‘per gene’ permutation test and a ‘pooled null’ permutation test. The inset magnifies the plot for small P-values. These results are for the real CEU/CHB data (not simulated data) with 10 000 randomly chosen genes to facilitate presentation. See Supplementary Figure 7 for the corresponding plots for other test statistics.

Fig. 5.

MLE estimates of π₁^μ using locfdr. A reference line is drawn at π₁^μ, the proportion of genes DE in the mean. The s-statistics were scaled by 1/0.55 and then treated as a t-distribution with 8 , 9 or 11 degrees of freedom for π₁^μ=0.01, 0.05, and 0.2, respectively.