On the importance of avoiding shortcuts in applying cognitive models to hierarchical data

Abstract

Psychological experiments often yield data that are hierarchically structured. A number of popular shortcut strategies in cognitive modeling do not properly accommodate this structure and can result in biased conclusions. To gauge the severity of these biases, we conducted a simulation study for a two-group experiment. We first considered a modeling strategy that ignores the hierarchical data structure. In line with theoretical results, our simulations showed that Bayesian and frequentist methods that rely on this strategy are biased towards the null hypothesis. Secondly, we considered a modeling strategy that takes a two-step approach by first obtaining participant-level estimates from a hierarchical cognitive model and subsequently using these estimates in a follow-up statistical test. Methods that rely on this strategy are biased towards the alternative hypothesis. Only hierarchical models of the multilevel data lead to correct conclusions. Our results are particularly relevant for the use of hierarchical Bayesian parameter estimates in cognitive modeling.

Keywords: Bayes factor; Cognitive models; Hierarchical Bayesian model; Statistical errors; Statistical test.

Fig. 1

Full hierarchical model. $\mathcal{N}$ denotes the normal prior, $\mathcal{U}$ denotes the uniform prior, and T(0,) indicates truncation at 0

Fig. 2

Non-hierarchical model. $\mathcal{N}$ denotes the normal prior distribution, $\mathcal{U}$ denotes the uniform prior, and T(0,) indicates truncation at 0

Fig. 3

Outcomes of the Bayesian analysis under the hierarchical and non-hierarchical Bayesian model for different numbers of simulated trials (K) and participants (N) for δ = 0. The scatterplot shows a comparison of log-Bayes factors for the hierarchical (BF_10H, y-axis) and non-hierarchical (BF_10NH, x-axis) Bayesian model. The gray diagonal line shows where log-Bayes factors should fall in the case of equality ($log{\text{BF}}_{10H}=log{\text{BF}}_{10\mathrm{NH}}$). The gray dotted lines indicate the indecision point where $log\text{BF}=1$. Histograms show the marginal distribution of the log-Bayes factors

Fig. 4

Outcomes of the Bayesian analysis under the hierarchical and non-hierarchical Bayesian model for different numbers of simulated trials (K) and participants (N) for δ = 1. The scatterplot shows a comparison of log-Bayes factors for the hierarchical (BF_10H, y-axis) and non-hierarchical (BF_10NH, x-axis) Bayesian model. Red asterisks indicate outliers (outliers are jittered to prevent visual overlap). The gray diagonal line shows where log-Bayes factors should fall in the case of equality ($log{\text{BF}}_{10H}=log{\text{BF}}_{10\mathrm{NH}}$). The gray dotted lines indicate the indecision point where $log\text{BF}=1$. Histograms show the marginal distribution of the log-Bayes factors

Fig. 5

Differences between log-Bayes factors under the hierarchical and non-hierarchical Bayesian model. Violin plots show the distribution of differences between absolute log-Bayes factors, $|log{\text{BF}}_{10H}|-|log{\text{BF}}_{10\mathrm{NH}}|$, for different numbers of simulated trials (K) and participants (N). Dashed horizontal lines indicate no difference in log-Bayes factors

Fig. 6

Posterior distribution of effect size δ under the hierarchical and non-hierarchical Bayesian model for different numbers of simulated trials (K) and participants (N). Distributions shown are the prior (light gray dashed lines) and quantile-averaged posterior distributions of δ under the hierarchical (H, black) and non-hierarchical model (NH, dark gray) for δ = 0 (left subplot) and δ = 1 (right subplot). The gray solid vertical line indicates the mean of the prior distribution and the black dashed vertical line shows the true value of δ

Fig. 7

Outcomes of the frequentist analysis for different numbers of simulated trials (K) and participants (N). Top row: t values for δ = 0 (left subplot) and δ = 1 (right subplot). Dotted lines show t = 0, dashed lines show the critical t value in a two-sided t test with α = .05, and red lines show the theoretical t value. Dots are true t values (TR; blue), t values from a hierarchical frequentist strategy (HF; green), non-hierarchical frequentist strategy (NF; grey), and two-step frequentist strategy (TF; orange); asterisks denote outliers (outliers are jittered to prevent visual overlap). Numbers at the bottom indicate the proportion of significant t values (out of 200 t tests). Bottom row: p values for δ = 0 (left subplot) and for δ = 1 (right subplot). Solid lines indicate p = .05. Dots are true p values (blue), p values from a hierarchical frequentist strategy (green), non-hierarchical strategy (grey), and two-step frequentist strategy (orange). Data points are jittered for improved visibility