Statistical evidence in psychological networks

Abstract

Psychometric network models have become increasingly popular in psychology and the social sciences as tools to explore multivariate data. In these models, constructs are represented as networks of observed variables, and researchers often interpret the presence or absence of edges as evidence for or against conditional associations between variables. However, the statistical evidence supporting these edges is rarely evaluated. Here we show that a large proportion of reported network findings is based on weak or inconclusive evidence. We reanalysed 293 networks from 126 published papers using a Bayesian approach that quantifies the evidence for each edge. Across the studies, one-third of edges showed inconclusive evidence (1/3 < inclusion Bayes factor (BF₁₀) < 3), about half showed weak evidence (BF₁₀ > 3 or BF₁₀ < 1/3) and fewer than 20% were strongly supported (BF₁₀ > 10 or BF₁₀ < 1/10). Networks based on relatively large sample sizes yielded more-robust results. Our study shows that networks are often supported by too little evidence from the data for the results to be reported with confidence, not meaning that the results are flawed but, rather, suggesting caution in interpreting individual edges.