Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Oct 17;46(5):497-502.
doi: 10.5993/AJHB.46.5.1.

Comparing Data Collected on Amazon's Mechanical Turk to National Surveys

Nabeel Qureshi  1 , Maria Edelen  2 , Lara Hilton  3 , Anthony Rodriguez  4 , Ron D Hays  5 , Patricia M Herman  1
Affiliations

Comparing Data Collected on Amazon's Mechanical Turk to National Surveys

Nabeel Qureshi et al. Am J Health Behav. .

Abstract

Objective: In this study, we examined the impact of a range of methods to improve data quality on the demographic and health status representativeness of Amazon Mechanical Turk (MTurk) samples.

Methods: We developed and field-tested a general survey of health on MTurk in 2017 among 5755 participants and 2021 among 6752 participants. We collected information on participant demographic characteristics and health status and implemented different quality checks in 2017 and 2021.

Results: Adding data quality checks generally improves the representativeness of the final MTurk sample, but there are persistent differences in mental health and pain conditions, age, education, and income between the MTurk population and the broader US population.

Conclusion: We conclude that data quality checks improve the data quality and representativeness.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest Statement

The authors have no relevant financial or non-financial interests to disclose.

Figures

Figure 1:
Figure 1:
Age distribution in MTurk compared to US population
See this image and copyright information in PMC

References

    1. Buhrmester M, Kwang T, Gosling SD. Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspect Psychol Sci. Jan 2011;6(1):3–5. doi: 10.1177/1745691610393980 - DOI - PubMed
    1. Mason W, Suri S. Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods. 2012/03/01 2012;44(1):1–23. doi: 10.3758/s13428-011-0124-6 - DOI - PubMed
    1. Mortensen K, Hughes TL. Comparing Amazon’s Mechanical Turk Platform to Conventional Data Collection Methods in the Health and Medical Research Literature. J Gen Intern Med. Apr 2018;33(4):533–538. doi: 10.1007/s11606-017-4246-0 - DOI - PMC - PubMed
    1. Walters K, Christakis DA, Wright DR. Are Mechanical Turk worker samples representative of health status and health behaviors in the U.S.? PLoS One. 2018;13(6):e0198835. doi: 10.1371/journal.pone.0198835 - DOI - PMC - PubMed
    1. Huff C, Tingley D. “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents. Research & Politics. 2015/07/01 2015;2(3):2053168015604648. doi: 10.1177/2053168015604648 - DOI