Comparison of seven methods for producing Affymetrix expression scores based on False Discovery Rates in disease profiling data

Abstract

Background: A critical step in processing oligonucleotide microarray data is combining the information in multiple probes to produce a single number that best captures the expression level of a RNA transcript. Several systematic studies comparing multiple methods for array processing have used tightly controlled calibration data sets as the basis for comparison. Here we compare performances for seven processing methods using two data sets originally collected for disease profiling studies. An emphasis is placed on understanding sensitivity for detecting differentially expressed genes in terms of two key statistical determinants: test statistic variability for non-differentially expressed genes, and test statistic size for truly differentially expressed genes.

Results: In the two data sets considered here, up to seven-fold variation across the processing methods was found in the number of genes detected at a given false discovery rate (FDR). The best performing methods called up to 90% of the same genes differentially expressed, had less variable test statistics under randomization, and had a greater number of large test statistics in the experimental data. Poor performance of one method was directly tied to a tendency to produce highly variable test statistic values under randomization. Based on an overall measure of performance, two of the seven methods (Dchip and a trimmed mean approach) are superior in the two data sets considered here. Two other methods (MAS5 and GCRMA-EB) are inferior, while results for the other three methods are mixed.

Conclusions: Choice of processing method has a major impact on differential expression analysis of microarray data. Previously reported performance analyses using tightly controlled calibration data sets are not highly consistent with results reported here using data from human tissue samples. Performance of array processing methods in disease profiling and other realistic biological studies should be given greater consideration when comparing Affymetrix processing methods.

Figure 1

Sensitivity results for colon and ovary data. Top row: number of significant probe sets at a range of FDR₀ values using the t-test statistic. Bottom row: number of significant probe sets at a range of FDR₀ values using the rank-sum statistic. The left column shows the results for colon data and the right column shows the results for ovary data.

Figure 2

FDR agreement between methods. The ratio of the number of probe sets with FDR₀ value below a given threshold in k or more of the seven methods to the number of probe sets with FDR₀ value below the threshold in at least one method was calculated for k = 3, 4, 5, 6, 7, and plotted against the FDR₀ threshold. Results are shown for the colon data (left column), the ovary data (right column), and for the t-test statistic (top row), and the rank-sum statistic (bottom row).

Figure 3

Calibration results for ovary and colon data. The threshold test statistic required to obtain a given FDR₀ for each method is plotted against the FDR₀ value. Results are shown for the colon data (left column), the ovary data (right column), and for the t-test statistic (top row), and the rank-sum statistic (bottom row).

Figure 4

Test statistics for ovary and colon data. For each of the seven processing methods, the number of probe sets exceeding a test statistic threshold t was calculated and plotted against log₂ t. Results are shown for the colon data (left column), the ovary data (right column), and for the t-test statistic (top row), and the rank-sum statistic (bottom row).

Figure 5

Sensitivity for detecting genes with at least 50% change in expression magnitude. The number of significant probe sets at a range of FDR₀ values is shown for analysis in which the test statistic is the t-statistic truncated to zero when the fold change is less than 50%.