Correction of unexpected distributions of P values from analysis of whole genome arrays by rectifying violation of statistical assumptions

Abstract

Background: Statistical analysis of genome-wide microarrays can result in many thousands of identical statistical tests being performed as each probe is tested for an association with a phenotype of interest. If there were no association between any of the probes and the phenotype, the distribution of P values obtained from statistical tests would resemble a Uniform distribution. If a selection of probes were significantly associated with the phenotype we would expect to observe P values for these probes of less than the designated significance level, alpha, resulting in more P values of less than alpha than expected by chance.

Results: In data from a whole genome methylation promoter array we unexpectedly observed P value distributions where there were fewer P values less than alpha than would be expected by chance. Our data suggest that a possible reason for this is a violation of the statistical assumptions required for these tests arising from heteroskedasticity. A simple but statistically sound remedy (a heteroskedasticity-consistent covariance matrix estimator to calculate standard errors of regression coefficients that are robust to heteroskedasticity) rectified this violation and resulted in meaningful P value distributions.

Conclusions: The statistical analysis of 'omics data requires careful handling, especially in the choice of statistical test. To obtain meaningful results it is essential that the assumptions behind these tests are carefully examined and any violations rectified where possible, or a more appropriate statistical test chosen.

Figure 1

The distribution of P values for regression coefficients with a neuro-cognitive outcome. The association of 237,152 probe log ratio values with a continuous neuro-cognitive outcome were obtained using linear regression controlling for gender. Figure 1 shows that there are more P values ≤ 0.05 than would be expected by chance and thus suggests that some probes are significantly associated with the neuro-cognitive outcome.

Figure 2

The distribution of P values after a simulation exercise using permutation of outcome values. Figure 2 shows there are no more P values ≤ 0.05 than would be expected by chance, resulting in a distribution similar to a Uniform distribution.

Figure 3

The distribution of P values for regression coefficients with a body composition outcome. Figure 3 shows that fewer P values ≤ 0.05 were obtained than would be expected by chance.

Figure 4

Regression residuals for a body composition outcome plotted against values predicted by the regression. Figure 4 shows the regression residuals increasing with predicted value (the points on the plot spread out in a fan shape), indicating heteroskedasticity.

Figure 5

The distribution of P values with a body composition outcome using robust standard errors. Figure 5 indicates that a number of probes are associated with the outcome of interest as there are more P values ≤ 0.05 than would be expected by chance.

Figure 6

Comparison of P values ≤ 0.01 using classical linear regression recalculated using robust standard errors. Figure 6 shows that the majority of these P values are still ≤ 0.01 after re-estimation.