Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2018 Mar 13;115(11):2607-2612.
doi: 10.1073/pnas.1708285114. Epub 2018 Mar 12.

Metastudies for robust tests of theory

Beth Baribault  1 , Chris Donkin  2 , Daniel R Little  3 , Jennifer S Trueblood  4 , Zita Oravecz  5 , Don van Ravenzwaaij  6 , Corey N White  7 , Paul De Boeck  8 , Joachim Vandekerckhove  9
Affiliations

Metastudies for robust tests of theory

Beth Baribault et al. Proc Natl Acad Sci U S A. .

Abstract

We describe and demonstrate an empirical strategy useful for discovering and replicating empirical effects in psychological science. The method involves the design of a metastudy, in which many independent experimental variables-that may be moderators of an empirical effect-are indiscriminately randomized. Radical randomization yields rich datasets that can be used to test the robustness of an empirical claim to some of the vagaries and idiosyncrasies of experimental protocols and enhances the generalizability of these claims. The strategy is made feasible by advances in hierarchical Bayesian modeling that allow for the pooling of information across unlike experiments and designs and is proposed here as a gold standard for replication research and exploratory research. The practical feasibility of the strategy is demonstrated with a replication of a study on subliminal priming.

Keywords: generalizability; many labs; metastudy; radical randomization; robustness.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Fig. 1.
Fig. 1.
Level-2 variability. Histograms of estimated effect sizes across microexperiments are split between masked (Left) and unmasked (Right) conditions and between microexperiments that support an effect (regular bars) versus no effect (inverted bars). Darker bars indicate stronger support with a Bayes factor of at least 10. A majority of microexperiments show support for the unmasked effect, but a similarly large number support no effect of the masked cue.
Fig. 2.
Fig. 2.
(Left) Microexperiments support an effect when participants are able to consciously identify the cue (square markers), but not otherwise (round markers). (Right) The data are split by subliminality. The facet “target center color” was varied over 13 possible levels, but the facet does not appear to moderate the effect of interest. That is, the effect appears robust against this facet. In both panels error bars show 99% credibility intervals. Solid square markers indicate strong evidence (Bayes factor >10) for a nonzero value. Solid round markers indicate strong evidence for a zero value. Empty markers indicate ambiguous evidence.
See this image and copyright information in PMC

References

    1. Pashler H, Wagenmakers EJ. Editor’s introduction to the special section on replicability in psychological science: A crisis of confidence? Perspect Psychol Sci. 2012;7:528–530. - PubMed
    1. Francis G. Publication bias and the failure of replication in experimental psychology. Psychon Bull Rev. 2012;19:975–991. - PubMed
    1. Guan M, Vandekerckhove J. A Bayesian approach to mitigation of publication bias. Psychon Bull Rev. 2016;23:74–86. - PubMed
    1. Rosenthal R. The file drawer problem and tolerance for null results. Psychol Bull. 1979;86:638–641.
    1. Vasishth S, Gelman A. 2017. The illusion of power: How the statistical significance filter leads to overconfident expectations of replicability. arXiv:1702.00556.

Publication types