MetaQC: objective quality control and inclusion/exclusion criteria for genomic meta-analysis

Abstract

Genomic meta-analysis to combine relevant and homogeneous studies has been widely applied, but the quality control (QC) and objective inclusion/exclusion criteria have been largely overlooked. Currently, the inclusion/exclusion criteria mostly depend on ad-hoc expert opinion or naïve threshold by sample size or platform. There are pressing needs to develop a systematic QC methodology as the decision of study inclusion greatly impacts the final meta-analysis outcome. In this article, we propose six quantitative quality control measures, covering internal homogeneity of coexpression structure among studies, external consistency of coexpression pattern with pathway database, and accuracy and consistency of differentially expressed gene detection or enriched pathway identification. Each quality control index is defined as the minus log transformed P values from formal hypothesis testing. Principal component analysis biplots and a standardized mean rank are applied to assist visualization and decision. We applied the proposed method to 4 large-scale examples, combining 7 brain cancer, 9 prostate cancer, 8 idiopathic pulmonary fibrosis and 17 major depressive disorder studies, respectively. The identified problematic studies were further scrutinized for potential technical or biological causes of their lower quality to determine their exclusion from meta-analysis. The application and simulation results concluded a systematic quality assessment framework for genomic meta-analysis.

Figure 1.

Example of IQC calculation and reverse transformation of P_IQC. (A) Three points (E2, E3 and E4) in the lower left represent homogeneous studies, and a point (E1) in the upper right is a heterogeneous study which has larger pair-wise distance to others. A heterogeneous study should have larger pair-wise distances with others. The IQC hypothesis setting compares (d₁₂,d₁₃,d₁₄) and (d₂₃,d₂₄,d₃₄) by Wilcoxon rank-sum test. (B) X- and Y-axes are P-values before and after applying the reverse transformation g. As a result, small P-values will be transformed to large pseudo P-values and vice versa.

Figure 2.

PCA biplots of QC measures in four examples. Each circled number represents the overall rank by SMR score of a study. Smaller numbers correspond to higher quality studies. (A) Seven brain cancer studies. (B) Nine prostate cancer studies. (C) Eight IPF studies. (D) Seventeen MDD studies.

Figure 3.

Marginal impacts on meta-analysis for DE genes detection. X-axis represents each study included cumulatively to a series of meta-analyses. The order of addition follows the SMR score in the Table 4. Y-axis represents the number of DE genes detected. (A) Brain cancer example under FDR = 0.5% threshold. (B) Prostate cancer example under FDR = 0.1% threshold. (C) IPF example under FDR = 0.1% threshold. (D) MDD example under P-value = 0.01 threshold.

Figure 4.

Marginal impacts on meta-analysis for enriched pathway detection. Similar to Figure 3, X-axis represents each study included cumulatively to a series of meta-analyses. The order of addition follows the SMR score in the Table 4. Y-axis represents the number of pathways detected. All examples are shown under FDR = 5% threshold. (A) Brain cancer example. (B) Prostate cancer example. (C) IPF example. (D) MDD example.

Figure 5.

Simulations showing effects of adding an irrelevant ‘spike-in’ study. Y-axis represents the value of 1-SMR. Greater Y-axis values correspond to better quality studies judged by MetaQC. X-axis represents the addition of an irrelevant ‘spike-in’ study. A set of irrelvant studies were obtained from one of the other three examples. ‘NA’ represents the original quality result without the spike-in simulation which is the same result as Table 4. A black asterisk represents the 1-SMR of the added irrelevant study. (A) Brain cancer studies with a spiked-in prostate cancer study. (B) Prostate cancer studies with a spiked-in brain cancer study. (C) IPF studies with a spiked-in brain cancer study. (D) MDD studies with a spiked-in brain cancer study.