US studies may overestimate effect sizes in softer research

Abstract

Many biases affect scientific research, causing a waste of resources, posing a threat to human health, and hampering scientific progress. These problems are hypothesized to be worsened by lack of consensus on theories and methods, by selective publication processes, and by career systems too heavily oriented toward productivity, such as those adopted in the United States (US). Here, we extracted 1,174 primary outcomes appearing in 82 meta-analyses published in health-related biological and behavioral research sampled from the Web of Science categories Genetics & Heredity and Psychiatry and measured how individual results deviated from the overall summary effect size within their respective meta-analysis. We found that primary studies whose outcome included behavioral parameters were generally more likely to report extreme effects, and those with a corresponding author based in the US were more likely to deviate in the direction predicted by their experimental hypotheses, particularly when their outcome did not include additional biological parameters. Nonbehavioral studies showed no such "US effect" and were subject mainly to sampling variance and small-study effects, which were stronger for non-US countries. Although this latter finding could be interpreted as a publication bias against non-US authors, the US effect observed in behavioral research is unlikely to be generated by editorial biases. Behavioral studies have lower methodological consensus and higher noise, making US researchers potentially more likely to express an underlying propensity to report strong and significant findings.

Keywords: publish or perish; questionable research practices; research bias; scientific misconduct; soft science.

Fig. 1.

Magnitude of effect sizes of primary studies relative to the summary effect size in their respective meta-analysis, partitioned by geographical origin of their corresponding author and by type of study. Size of circle is proportional to study size, measured by the SE. For illustration purposes, here size is equal to ln(2/SE). The value of 0 corresponds to a perfect matching between primary study and the summary effect size calculated from the study’s meta-analysis, using a random effects model. The range of values was limited to −1 +1 to show more details. The complete range is given in Fig. S1. AS, China, Japan, South Korea, Taiwan, Singapore, and India; EU15, European Union-15 countries; o, all other countries; US, United States.

Fig. 2.

Deviation score (i.e., double square root-transformed absolute value of data points shown in Fig. 1) plotted against study size [log(SE)], partitioned by type of study. The value of 0 corresponds to a perfect matching between primary study and the summary effect size calculated from the study’s meta-analysis, using a random-effects model. Size of circle is inversely proportional to a study’s order of appearance (chronological order of publication) in its meta-analysis: larger circles indicate first studies. Nonweighted regression lines were added to illustrate trends for studies of different geographical origin (based on corresponding author). Discipline classification follows the Web of Science system and is based on journal of publication. AS, China, Japan, South Korea, Taiwan, Singapore, and India; EU15, European Union-15 countries; US, United States.