Toward Cumulative Cognitive Science: A Comparison of Meta-Analysis, Mega-Analysis, and Hybrid Approaches

Abstract

There is increasing interest in cumulative approaches to science, in which instead of analyzing the results of individual papers separately, we integrate information qualitatively or quantitatively. One such approach is meta-analysis, which has over 50 years of literature supporting its usefulness, and is becoming more common in cognitive science. However, changes in technical possibilities by the widespread use of Python and R make it easier to fit more complex models, and even simulate missing data. Here we recommend the use of mega-analyses (based on the aggregation of data sets collected by independent researchers) and hybrid meta- mega-analytic approaches, for cases where raw data are available for some studies. We illustrate the three approaches using a rich test-retest data set of infants' speech processing as well as synthetic data. We discuss advantages and disadvantages of the three approaches from the viewpoint of a cognitive scientist contemplating their use, and limitations of this article, to be addressed in future work.

Keywords: cumulative science; data simulation; fixed effects; mega-analyses; meta-analyses; open science; random effects.

Figure 1.

Fixed effects in the analysis of (a) effect size in the first day of testing and (b) the test-retest correlation, according to three estimation methods: A meta-analysis (meta); a regression with the true individual-level data (mega); and a regression where all individual-level data points have been generated (hybrid). Error bars indicate standard errors. In the hybrid analysis (hybrid), standard errors have been calculated considering both variation within studies (SE of fixed effects in each simulation) and between studies (SE between means of each simulation).

Figure 2.

Random intercepts per study in the analysis of effects in the first day of testing, representing deviations from the fixed effects in Figure 1a, according to three estimation methods: A meta-analysis (meta); a regression with the true individual-level data (mega); and a regression where all individual-level data points have been generated (hybrid). Error bars indicate standard errors. In the hybrid analysis (hybrid), standard errors have been calculated considering both variation within studies (SE of random effects in each simulation) and between studies (SE between means of each simulation).

Figure 3.

Random slopes per study in the analysis of test-retest correlation, representing deviations from the fixed effects in Figure 1b, according to three estimation methods: A meta-analysis (meta); a regression with the true individual-level data (mega); and a regression where all individual-level data points have been generated (hybrid). Error bars indicate standard errors. In the hybrid analysis (hybrid), standard errors have been calculated considering both variation within studies (SE of random effects in each simulation) and between studies (SE between means of each simulation).

Figure 4.

Fixed effects for synthetic data in the analysis of effect size in the first day of testing (a) and of the correlation size across testing days (b), according to two estimation methods: A meta-analysis (meta); a regression with the true individual-level data (mega). In (a), we plot calculated d as a function of true correlation r in the data generation, and each horizontal panel represents different values of true effect size d. In (b), we plot calculated r as a function of true effect size d in data generation, and each horizontal panel represents different values of true correlation r. Error bars represent standard errors for the fitted values. Horizontal dotted lines represent the true value of the parameter (d at the top, r at the bottom).

Figure 5.

Random intercepts per study for synthetic data in the analysis of effects in the first day of testing, according to two estimation methods: A meta-analysis (meta); a regression with the true individual-level data (mega). In each panel, we plot calculated d as a function of the true values of d and r in the data generation. Error bars represent standard errors for the fitted values.

Figure 6.

Random slopes per study for synthetic data in the analysis of the correlation between days of testing, according to two estimation methods: A meta-analysis (meta); a regression with the true individual-level data (mega). In each panel, we plot calculated d as a function of the true values of d and r in the data generation. Error bars represent standard errors for the fitted values.