Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Dec 30;57(1):33.
doi: 10.3758/s13428-024-02573-3.

How do manipulation checks interfere with the inference of causal relationships?

Yuhwa Han  1 , Wooyeol Lee  2
Affiliations

How do manipulation checks interfere with the inference of causal relationships?

Yuhwa Han et al. Behav Res Methods. .

Abstract

This study investigates the performance of mediation analyses, including manipulation check variables, in experimental studies where manipulated psychological attributes are independent variables. We simulated the level of manipulation intensities and measurement errors of the manipulation check variable to test the validity of the analytic practice. Our results showed that when manipulation is successful and measurement error is low, mediation analyses with the manipulation check variable revealed an unstable path coefficient and standard error. Moreover, many of the detected indirect effects were inconsistent mediation situations. However, when individual differences in psychological attributes remained within the condition (low manipulation intensity) and the manipulation check variable contained low measurement error, the indirect effect indicated the validity of the manipulation. We discuss the implications of our findings for the use of manipulation checks in experimental research.

Keywords: Manipulation check; Manipulation intensity; Measurement error; Mediation analysis; Multicollinearity.

PubMed Disclaimer

Conflict of interest statement

Declarations. Conflicts of interest/Competing interests: The authors declare no competing interests. Ethics approval: Not applicable. Consent to participate: Not applicable. Consent for publication: Not applicable.

Figures

Fig. 1
Fig. 1
Distribution of the hypothetical psychological attribute T (the x-axis is an arbitrary unit)
Fig. 2
Fig. 2
Relationships between constructs. a The surface relationship between the independent variable (X) and the dependent variable (Y). b Path diagram when the X→Y relationship is significant and X is a valid manipulation of the psychological attribute T. c Path diagram when the X→Y relationship is significant, but the relationship between T and Y is not true. d Path diagram describing another situation where the manipulation is not valid (adapted and modified from Lench et al., 2014)
Fig. 3
Fig. 3
Detection rate of the indirect effects under the simulation conditions in Case 1
Fig. 4
Fig. 4
Detection rate of indirect effects under the simulation conditions in Case 2
Fig. 5
Fig. 5
Detection rate of indirect effects under the simulation conditions in Case 3
Fig. 6
Fig. 6
Detection rate of indirect effects under the simulation conditions in Case 4
See this image and copyright information in PMC

References

    1. Alin, A. (2010). Multicolliearity. WIREs. Computational Statistics,2(3), 370–374. 10.1002/wics.84 - DOI
    1. Aronson, E., & Carlsmith, J. M. (1968). Experimentation in social psychology. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.), The handbook of social psychology, 2(2), 1–79.
    1. Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic and statistical considerations. Journal of Personality and Social Psychology,51, 1173–1182. 10.1037/0022-3514.51.6.1173 - DOI - PubMed
    1. Campbell, D. T., & Cook, T. D. (1979). Quasi-experimentation. Rand Mc-Nally.
    1. Carlsmith, J. M., Ellswarth, P. C., & Aronson, E. (1976). Methods of research in social psychology. Addison-Wesley Publishing Company.

Publication types

LinkOut - more resources