A meta-analysis of procedures to change implicit measures

Abstract

Using a novel technique known as network meta-analysis, we synthesized evidence from 492 studies (87,418 participants) to investigate the effectiveness of procedures in changing implicit measures, which we define as response biases on implicit tasks. We also evaluated these procedures' effects on explicit and behavioral measures. We found that implicit measures can be changed, but effects are often relatively weak (|ds| < .30). Most studies focused on producing short-term changes with brief, single-session manipulations. Procedures that associate sets of concepts, invoke goals or motivations, or tax mental resources changed implicit measures the most, whereas procedures that induced threat, affirmation, or specific moods/emotions changed implicit measures the least. Bias tests suggested that implicit effects could be inflated relative to their true population values. Procedures changed explicit measures less consistently and to a smaller degree than implicit measures and generally produced trivial changes in behavior. Finally, changes in implicit measures did not mediate changes in explicit measures or behavior. Our findings suggest that changes in implicit measures are possible, but those changes do not necessarily translate into changes in explicit measures or behavior. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Figure 1.

PRISMA diagram of our data collection process (adapted from Moher et al., 2009). * This is a conservative estimate of the total number of records, as articles retrieved through direct requests and the unpublished meta-analysis that were excluded from the meta-analysis were not tracked systematically. ** We do not have a complete breakdown of reasons for excluding records. However, we recoded a random 10% (N = 486) of the records for reliability coding and provided exclusion reasons for those records. For detailed information about the results of this coding, see https://osf.io/6ex3n/

Figure 2.

Network plot of procedures included in the meta-analysis. The radius of the category circles = the number of procedures in that category, line width = the number of samples in which a pair of conditions were directly compared.

Figure 3.

Forest plot of the comparisons between each procedure and a neutral procedure. k gives the number of studies that directly (or indirectly, listed in parentheses) compare the listed procedure and a neutral procedure. g gives the estimated standardized mean difference and its 95% CI. Higher effect sizes reflect greater increases in the implicit measure relative to a neutral procedure.

Figure 4.

Moderation analyses. k gives the number of studies that directly (or indirectly, listed in parentheses) compare the listed procedure and a neutral procedure for the displayed levels of the moderator. “Difference” represents the difference between the two moderator levels and its 95% CI. Higher effect sizes reflect greater increases in implicit measures compared to a neutral procedure. Where there was not enough data in one of the moderator levels for estimation, the overall model estimate is shown instead.

Figure 5.

Forest plot of the consistency between effects on implicit and explicit measures. g gives the implicit and explicit estimates; g_I - g_E gives their difference. k gives the number of studies with implicit and explicit measures that directly (or indirectly, listed in parentheses) compare the listed procedure and a neutral procedure. “χ²” gives the 1 df Wald χ² test of the implicit-explicit difference, and “p” gives its p-value.

Figure 6.

Indirect effects (in the conventional mediation framework, the effect ab) of procedures on explicit measures through changes in implicit measures. k gives the number of studies that directly (or indirectly, listed in parentheses) compare the listed procedure and a neutral procedure.

Figure 7.

Forest plot of the consistency between effects on implicit and behavioral measures. g gives the implicit and behavioral estimates; g_I - g_B gives their difference. k gives the number of studies with implicit and behavioral measures that directly (or indirectly, listed in parentheses) compare the listed procedure and a neutral procedure. “χ²” gives the 1 df Wald χ² test of the implicit-behavioral difference, and “p” gives its p-value.

Figure 8.

Indirect effects (in the conventional mediation framework, the effect ab) of procedures on behavioral measures through changes in implicit measures. k gives the number of studies that directly (or indirectly, listed in parentheses) compare the listed procedure and a neutral procedure.

Figure 9.

Comparison-adjusted funnel plots of effect sizes vs standard errors for implicit, explicit, and behavioral measures. Positive numbers are more extreme relative to the meta-analytic comparison a study contributes to and negative numbers less extreme. The red line represents the fit from a mixed-effects regression; a line that departs from the vertical suggests the presence of small-study bias.

Figure 10.

Relationship between publication year and effect sizes on implicit measures. Larger points represent effect sizes that are estimated with greater precision. Only direct comparisons between each listed procedure and a neutral procedure are shown as points.