Underpowered samples, false negatives, and unconscious learning

Abstract

The scientific community has witnessed growing concern about the high rate of false positives and unreliable results within the psychological literature, but the harmful impact of false negatives has been largely ignored. False negatives are particularly concerning in research areas where demonstrating the absence of an effect is crucial, such as studies of unconscious or implicit processing. Research on implicit processes seeks evidence of above-chance performance on some implicit behavioral measure at the same time as chance-level performance (that is, a null result) on an explicit measure of awareness. A systematic review of 73 studies of contextual cuing, a popular implicit learning paradigm, involving 181 statistical analyses of awareness tests, reveals how underpowered studies can lead to failure to reject a false null hypothesis. Among the studies that reported sufficient information, the meta-analytic effect size across awareness tests was d z = 0.31 (95 % CI 0.24-0.37), showing that participants' learning in these experiments was conscious. The unusually large number of positive results in this literature cannot be explained by selective publication. Instead, our analyses demonstrate that these tests are typically insensitive and underpowered to detect medium to small, but true, effects in awareness tests. These findings challenge a widespread and theoretically important claim about the extent of unconscious human cognition.

Keywords: Contextual cuing; False negatives; Implicit learning; Null hypothesis Significance testing· Statistical power.

Fig. 1

Panel A shows a sequence of search displays as used in standard contextual cuing experiments. Participants are instructed to search for a T-shaped target among a series of L-shaped distractors. Some search displays are regularly repeated during training, whilst others are new, unrepeated (random) displays. Panel B shows the typical pattern of results: Participants become faster at finding the target among the distractors in repeated displays

Fig. 2

Contextual cuing experiments sorted by the number of trials of the awareness test (top panel) or by sample size (lower panel). Black bars denote statistical contrasts with significant results. Red bars denote statistical contrasts with marginally significant results

Fig. 3

Results of a simulation exploring the size difference between hit rate and false alarm rate depending on the number of patterns learned by the participant. See the main text for more details. Error bars denote standard errors of the means across simulations

Fig. 4

Ninety-five percent confidence intervals (CIs) of a subset of experiments contrasting hit rate versus false alarm rate in recognition tests. Given the heterogeneity of the studies included in the figure, Q(26) = 43.73, p = .016, the meta-analytic mean and CI shown in the last row were computed using a random effects model

Fig. 5

Histogram of the logarithmic Bayes Factors (BF ₁₀’s) included in the meta-analysis. Positive values indicate support for the alternative hypothesis (awareness) and negative values indicate support for the null hypothesis (unawareness). The inset depicts a scatterplot of effect sizes (Cohen’s d) and logarithmic BF ₁₀’s with the best fitting quadratic function

Fig. 6

Contextual cuing (msec) plotted against awareness (recognition hits minus false alarms, expressed in terms of effect size d) in 1,000 simulated participants. Mean contextual cuing across the entire sample is 100 msec (rightmost vertical line), while that in the subset of simulated participants scoring at or below d = 0 (open circles) is approximately 70 msec (dotted vertical line)

Fig. 7

Impact factor of journals that published papers mentioning or not mentioning “implicitness” in the title