Eleven years of student replication projects provide evidence on the correlates of replicability in psychology

Abstract

Cumulative scientific progress requires empirical results that are robust enough to support theory construction and extension. Yet in psychology, some prominent findings have failed to replicate, and large-scale studies suggest replicability issues are widespread. The identification of predictors of replication success is limited by the difficulty of conducting large samples of independent replication experiments, however: most investigations reanalyse the same set of $170\mathrm{replications}$. We introduce a new dataset of 176 replications from students in a graduate-level methods course. Replication results were judged to be successful in 49% of replications; of the 136 where effect sizes could be numerically compared, 46% had point estimates within the prediction interval of the original outcome (versus the expected 95%). Larger original effect sizes and within-participants designs were especially related to replication success. Our results indicate that, consistent with prior reports, the robustness of the psychology literature is low enough to limit cumulative progress by student investigators.

Keywords: cognitive psychology; large-scale replication project; metascience; pedagogical replication; replication; social psychology.

Figure 1.

Of the 210 projects conducted for the class, 176 are included in our analysis, after excluding reproducibility projects (with no new data collection), non-experimental replications and missing projects. Of the 176, 136 report sufficient information to calculate prediction intervals, and 112 report enough to calculate standardized effect sizes.

Figure 2.

Relationship between effect size of the original study, effect size of the replication study and subjective replication success rating, for those studies where effect size was applicable (N = 112).

Figure 3.

Distribution of subjective replication scores within categories. Bar heights are counts of studies.

Figure 4.

Correlations between predictor variables. See Methods for descriptions of how each variable was coded.

Figure 5.

Odds ratios and corresponding 95% credible interval (CrI) on the likelihood of having a higher subjective replication score as a function of the independent variable. Estimates from a model of all original-replication pairs (N = 176) are shown in blue, and from a model of all pairs with full statistical information (N = 112) are shown in red. A value of 1 indicates no association, greater than 1 indicates an association with higher replication scores and less than 1 an association with lower replication scores.