Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2017 Aug 8:8:1359.
doi: 10.3389/fpsyg.2017.01359. eCollection 2017.

Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations

Melissa G Keith  1 , Louis Tay  1 , Peter D Harms  2
Affiliations
Review

Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations

Melissa G Keith et al. Front Psychol. .

Abstract

Amazon Mechanical Turk (MTurk) is becoming a prevalent source of quick and cost effective data for organizational research, but there are questions about the appropriateness of the platform for organizational research. To answer these questions, we conducted an integrative review based on 75 papers evaluating the MTurk platform and 250 MTurk samples used in organizational research. This integrative review provides four contributions: (1) we analyze the trends associated with the use of MTurk samples in organizational research; (2) we develop a systems perspective (recruitment system, selection system, and work management system) to synthesize and organize the key factors influencing data collected on MTurk that may affect generalizability and data quality; (3) within each factor, we also use available MTurk samples from the organizational literature to analyze key issues (e.g., sample characteristics, use of attention checks, payment); and (4) based on our review, we provide specific recommendations and a checklist for data reporting in order to improve data transparency and enable further research on this issue.

Keywords: mechanical turk; meta-analysis; research design; review; sampling.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Studies publishing MTurk samples in the organizational sciences 2012–2015.
Figure 2
Figure 2
Systems perspective of MTurk.
See this image and copyright information in PMC

References

    1. Alonso O., Mizzaro S. (2012). Using crowdsourcing for TREC relevance assessment. Inform. Process. Manage. 48, 1053–1066. 10.1016/j.ipm.2012.01.004 - DOI
    1. Amir O., Rand D. G., Gal Y. K. (2012). Economic games on the internet: the effect of $1 stakes. PLoS ONE 7:e311461. 10.1371/journal.pone.00311461 - DOI - PMC - PubMed
    1. Antin J., Shaw A. (2012). Social desirability bias and self-reports of motivation: A study of Amazon Mechanical Turk in the US and India, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. (Austin, TX: ), 2925–2934.
    1. Arditte K. A., Cek D., Shaw A. M., Timpano K. R. (2016). The importance of assessing clinical phenomena in Mechanical Turk research. Psychol. Assess. 28, 684–691. 10.1037/pas0000217 - DOI - PubMed
    1. Barber L. K., Barnes C. M., Carlson K. D. (2013). Random and systematic error effects of insomnia on survey behavior. Organ. Res. Methods 16, 616–649. 10.1177/1094428113493120 - DOI

LinkOut - more resources