Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Abstract

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.

Keywords: Analytical heterogeneity; Many-analyst; Metascience; Replication crisis; Reproducibility.

Fig. 1

Schematic of research process showing recruited analyst and reviewer contributions in orange and core team contributions in blue. Items that are crossed out were preregistered but could not be conducted. Items with a greyed background were added as exploratory analyses after preregistration

Fig. 2

Forest plots of meta-analytic estimated standardized effect sizes (Z _r, blue triangles) and their 95% confidence intervals for each effect size included in the meta-analysis model. A Blue tit analyses: Points where Z _r are less than 0 indicate analyses that found a negative relationship between sibling number and nestling growth. B Eucalyptus analyses: Points where Z _r are less than 0 indicate a negative relationship between grass cover and Eucalyptus seedling success. The meta-analytic mean effect size is denoted by a black circle and a dashed vertical line, with error bars also representing the 95% confidence interval. The solid black vertical line demarcates effect size of 0, indicating no relationship between the test variable and the response variable. Note that the Eucalyptus plot omits one extreme outlier with the value of −4.47 (Supplementary Material A, Figure A.2) in order to standardize the x-axes on these two panels

Fig. 3

Forest plot of meta-analytic estimated out-of-sample predictions. A Standardized (z-score) blue tit out-of-sample predictions, y _i. B Response-scale (stem counts) Eucalyptus out-of-sample predictions. Triangles represent individual estimates. Circles represent the meta-analytic mean for each prediction scenario. Dark-blue points correspond to y₂₅ scenario, medium-blue points correspond to the y₅₀ scenario, while light blue points correspond to the y₇₅ scenario. Error bars are 95% confidence intervals. Note that, for the Eucalyptus analysis, outliers (observations more than 3 $\text{SD}$ above the mean) have been removed prior to model fitting and do not appear on this figure. The x-axis is truncated to approximately 140, and thus some error bars are incomplete. See Supplementary Material B, Figure B.6 for full figure

Fig. 4

Violin plot of Box-Cox transformed deviation from meta-analytic mean ${\overline{Z}}_r$ as a function of categorical peer rating. Grey points for each rating group denote model-estimated marginal mean deviation, and error bars denote 95%CI of the estimate. A Blue tit dataset. B Eucalyptus dataset

Fig. 5

Fitted model of the Box-Cox-transformed deviation score (deviation in effect size from meta-analytic mean) as a function of the mean Sorensen’s index showing distinctiveness of the set of predictor variables. Grey ribbons on predicted values are 95% CI’s. A blue tit dataset. B Eucalyptus dataset

Fig. 6

Violin plot of mean Box-Cox transformed deviation from meta-analytic mean as a function of random-effects inclusion in Eucalyptus analyses. White point for each group of analyses denotes model-estimated marginal mean deviation, and error bars denote 95% CI of the estimate