Quantifying the Difference between Active and Passive Control Groups in Cognitive Interventions Using two Meta-Analytical Approaches

Abstract

Despite promising reports of broad cognitive benefit in studies of cognitive training, it has been argued that the reliance of many studies on no-intervention control groups (passive controls) make these reports difficult to interpret because placebo effects cannot be ruled out. Although researchers have recently been trying to incorporate more active controls, in which participants engage in an alternate intervention, previous work has been contentious as to whether this actually yields meaningfully different results. To better understand the influence of passive and active control groups on cognitive interventions, we conducted two meta-analyses to estimate their relative effect sizes. While the first one broadly surveyed the literature by compiling data from 34 meta-analyses, the second one synthesized data from 42 empirical studies that simultaneously employed both types of controls. Both analyses showed no meaningful performance difference between passive and active controls, suggesting that current active control placebo paradigms might not be appropriately designed to reliably capture these non-specific effects or that these effects are minimal in this literature.

Keywords: Cognitive Training; Experimental Confounds; Hawthorne Effects; Meta-Analysis; Placebo Effects.

Figure 1.

Flow Chart of Study Extraction Process.

Figure 2.. Meta-Meta-Analysis of Experimental vs. Active Control Comparisons.

Across meta-analyses, cognitive training studies using active control groups yield an overall effect size of d=0.308. Positive effect sizes favor experimental groups.

Figure 3.. Meta-Meta-Analysis of Experimental vs. Passive Control Comparisons.

Across meta-analyses, cognitive training studies using passive control groups yield an overall effect size of d=0.344. Positive effect sizes favor experimental groups.

Figure 4.. Meta-Meta-Analysis of Studies with Active vs. Passive Controls.

Across meta-analyses, cognitive training studies with passive controls yield an effect size that is d=0.030 larger than studies with active controls. Positive effect sizes favor studies with passive controls, and suggest the possibility, but not the necessity, of placebo-like effects. However, the difference is only marginally significant (p=0.052), and Bayesian statistics provide no support for the alternative hypothesis that any difference truly exists (BF₁₀=0.859).

Figure 5.. Meta-Analysis of Experimental/Active Control Comparisons.

Within our sample of double-controlled studies, the effect size of cognitive training on objective cognitive tests when compared to active controls is g=0.250. Outcomes that were specifically trained were excluded from analysis; thus this effect size only reflects transfer to untrained tasks.

Figure 6.. Meta-Analysis of Experimental/Passive Control Comparisons.

Within our sample of double-controlled studies, the effect size of cognitive training on objective cognitive tests when compared to active controls is g=0.309. Outcomes that were specifically trained were excluded from analysis; thus this effect size only reflects transfer to untrained tasks.

Figure 7.. Meta-Analysis of Active/Passive Control Comparisons.

Within our sample of double-controlled studies, the within-study performance difference between active and passive control groups is not significant (g=0.058), and Bayesian statistics support the null hypothesis (BF01=12.046). Outcomes that were specifically trained were excluded from analysis.

Figure 8.. Funnel Plots from Comparisons of Experimental with Passive and Active Control Groups (Double-Controlled Studies).

No asymmetry was statistically detectable in the funnel plots, neither for the comparison of experimental groups with passive controls (left) nor the comparison with active controls (right). More critically, the degree of asymmetry, though non-significant, is similar between both comparisons, suggesting that if bias does exist, it does not systematically affect one type of control group over the other.