Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Oct;51(5):2022-2038.
doi: 10.3758/s13428-019-01273-7.

Online panels in social science research: Expanding sampling methods beyond Mechanical Turk

Jesse Chandler  1 , Cheskie Rosenzweig  2 , Aaron J Moss  3 , Jonathan Robinson  4 , Leib Litman  5
Affiliations

Online panels in social science research: Expanding sampling methods beyond Mechanical Turk

Jesse Chandler et al. Behav Res Methods. 2019 Oct.

Abstract

Amazon Mechanical Turk (MTurk) is widely used by behavioral scientists to recruit research participants. MTurk offers advantages over traditional student subject pools, but it also has important limitations. In particular, the MTurk population is small and potentially overused, and some groups of interest to behavioral scientists are underrepresented and difficult to recruit. Here we examined whether online research panels can avoid these limitations. Specifically, we compared sample composition, data quality (measured by effect sizes, internal reliability, and attention checks), and the non-naivete of participants recruited from MTurk and Prime Panels-an aggregate of online research panels. Prime Panels participants were more diverse in age, family composition, religiosity, education, and political attitudes. Prime Panels participants also reported less exposure to classic protocols and produced larger effect sizes, but only after screening out several participants who failed a screening task. We conclude that online research panels offer a unique opportunity for research, yet one with some important trade-offs.

Keywords: Data collection; Mechanical Turk; Online experimentation; Prime panels.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
Estimates of the height of Mount Everest as a function of sample and anchor
Fig. 2
Fig. 2
Opposition to abortion as a function of sample and condition
Fig. 3
Fig. 3
Support for political equality as a function of sample and whether or not equality was defined
See this image and copyright information in PMC

References

    1. American National Election Studies, Stanford University, & University of Michigan. (2016). American National Election Study: 2016 pilot study. Ann Arbor, MI: Inter-University Consortium for Political and Social Research. Retrieved from 10.3886/ICPSR36390.v1
    1. Ballew, M. T., Goldberg, M. H., Rosenthal, S. A., Gustafson, A., & Leiserowitz, A. (2019). Systems thinking as a pathway to global warming beliefs and attitudes through an ecological worldview. Proceedings of the National Academy of Sciences, 116(17), 8214-8219. - PMC - PubMed
    1. Berinsky AJ, Huber GA, Lenz GS. Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis. 2012;20:351–368. doi: 10.1093/pan/mpr057. - DOI
    1. Bialek M, Pennycook G. The cognitive reflection test is robust to multiple exposures. Behavior Research Methods. 2018;50:1953–1959. doi: 10.3758/s13428-017-0963-x. - DOI - PubMed
    1. Buhrmester M, Kwang T, Gosling SD. Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science. 2011;6:3–5. doi: 10.1177/1745691610393980. - DOI - PubMed

LinkOut - more resources