Can I trust this paper?

Abstract

After a decade of data falsification scandals and replication failures in psychology and related empirical disciplines, there are urgent calls for open science and structural reform in the publishing industry. In the meantime, however, researchers need to learn how to recognize tell-tale signs of methodological and conceptual shortcomings that make a published claim suspect. I review four key problems and propose simple ways to detect them. First, the study may be fake; if in doubt, inspect the authors' and journal's profiles and request to see the raw data to check for inconsistencies. Second, there may be too little data; low precision of effect sizes is a clear warning sign of this. Third, the data may not be analyzed correctly; excessive flexibility in data analysis can be deduced from signs of data dredging and convoluted post hoc theorizing in the text, while violations of model assumptions can be detected by examining plots of observed data and model predictions. Fourth, the conclusions may not be justified by the data; common issues are inappropriate acceptance of the null hypothesis, biased meta-analyses, over-generalization over unmodeled variance, hidden confounds, and unspecific theoretical predictions. The main takeaways are to verify that the methodology is robust and to distinguish between what the actual results are and what the authors claim these results mean when citing empirical work. Critical evaluation of published evidence is an essential skill to develop as it can prevent researchers from pursuing unproductive avenues and ensure better trustworthiness of science as a whole.

Keywords: Power; Replication; Research integrity; Statistics.

Fig. 1

Model fit can be checked by plotting the observations (black points) together with fitted values (blue lines) and model predictions (blue points). Linear regression with a fixed standard deviation in this case underfits the data, whereas a Generalized Additive Model with spline smoothing captures the underlying sinusoidal trend perfectly. If only 10 datapoints are sampled from the same generative process, LOESS smoothing with a fixed span of 0.5 overfits the data. Inference from severely underfit or overfit models is highly suspect. Note that Pearson’s correlation assumes a linear relationship between variables, so it would be meaningless to report Pearson’s r between these two variables

Fig. 2

A simulation demonstrating some perils of inappropriate hypothesis testing in combination with insufficient data. It is tempting to interpret the non-significant p-value or the 95% credible interval (CI) that includes zero as evidence of no effect in group 2, but these are not valid ways to prove the null hypothesis (Bayes factors or equivalence testing could be used instead). Note also that the power is just 18% and the CIs are very wide, suggesting that there is simply too little data to estimate the effects in both groups with any certainty. If we specifically want to compare the effect in groups 1 and 2, we need to test for a treatment x group interaction or obtain the posterior distribution of the difference in the effect between groups, as shown above. NHST = null-hypothesis significance testing. Code: https://osf.io/n7r2y/

Fig. 3

Meta-analyses can reach overly optimistic conclusions if studies with significant and large effects are more likely to be published. Salami slicing a single dataset into multiple publications and including fraudulent data can further skew the results. Source of images: https://clipart-library.com