Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2014 Sep 1;84(3):328-364.
doi: 10.3102/0034654313500826.

Meta-Analysis With Complex Research Designs: Dealing With Dependence From Multiple Measures and Multiple Group Comparisons

Nancy Scammacca  1 , Greg Roberts  2 , Karla K Stuebing  3
Affiliations

Meta-Analysis With Complex Research Designs: Dealing With Dependence From Multiple Measures and Multiple Group Comparisons

Nancy Scammacca et al. Rev Educ Res. .

Abstract

Previous research has shown that treating dependent effect sizes as independent inflates the variance of the mean effect size and introduces bias by giving studies with more effect sizes more weight in the meta-analysis. This article summarizes the different approaches to handling dependence that have been advocated by methodologists, some of which are more feasible to implement with education research studies than others. A case study using effect sizes from a recent meta-analysis of reading interventions is presented to compare the results obtained from different approaches to dealing with dependence. Overall, mean effect sizes and variance estimates were found to be similar, but estimates of indexes of heterogeneity varied. Meta-analysts are advised to explore the effect of the method of handling dependence on the heterogeneity estimates before conducting moderator analyses and to choose the approach to dependence that is best suited to their research question and their data set.

Keywords: heterogeneity analysis; meta-analysis; statistical dependence.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Scatterplot of effect size by number of measures used in a study.
Figure 2
Figure 2
Scatterplot of effect size by number of standardized measures used in a study.
Figure 3
Figure 3
Scatterplot of effect sizes by number of groups in a study using all types of measures.
Figure 4
Figure 4
Scatterplot of effect sizes by number of groups in a study using standardized measures only.
See this image and copyright information in PMC

References

    1. Ahn S, Ames AJ, Myers ND. A review of meta-analyses in education: Methodological strengths and weaknesses. Review of Educational Research. 2012;82:436–476. doi: 10.3102/0034654312458162. - DOI
    1. American Psychological Association. Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist. 2008;63:839–851. doi: 10.1037/0003-066X.63.9.839. - DOI - PMC - PubMed
    1. Becker BJ. Multivariate meta-analysis. In: Tinsley HA, Brown SD, editors. Handbook of applied multivariate statistics and mathematical modeling. San Diego, CA: Academic Press; 2000. pp. 499–525. - DOI
    1. Berkeley S, Scruggs TE, Mastropieri MA. Reading comprehension instruction for students with learning disabilities, 1995–2006: A meta-analysis. Remedial and Special Education. 2010;31:423–436. doi: 10.1177/0741932509355988. - DOI
    1. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Complex data structures: Overview. In: Borenstein M, Hedges LV, Higgins JPT, Rothstein HR, editors. Introduction to meta-analysis. Chichester, England: John Wiley; 2009a. pp. 215–216.

LinkOut - more resources