Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research

Abstract

Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.

Figure 1. Congruent and Incongruent RTs, Error Rates and Individual Stroop Scores by Mean RT.

A. Mean RTs and error rates for congruent and incongruent Stroop items with standard error bars. B. Individual subject Stroop difference scores (incongruent-congruent) plotted as a function of individual subject mean RTs.

Figure 2. Repeat and Switch RTs, Error Rates and Individual Switch costs by mean RT.

A. Mean RTs and error rates for task repeat and switch trials with standard error bars. B. Individual subject switch costs (switch-repeat) plotted as a function of individual subject mean RTs.

Figure 3. Compatible and Incompatible RTs, Error Rates and Individual Flanker Scores by Mean RT.

A. Mean RTs and error rates for compatible and incompatible flanker items with standard error bars. B. Individual subject flanker scores (incompatible-compatible) plotted as a function of individual subject mean RTs.

Figure 4. Compatible and Incompatible RTs, Error Rates, and Individual Simon Scores by Mean RT.

A. Mean RTs and error rates for compatible and incompatible Simon trials with standard error bars. B. Individual subject Simon scores (incompatible-compatible) plotted as a function of individual subject mean RTs.

Figure 5. Visual Cuing: Cued and Uncued Mean RTs as a function of CSTOA.

Mean RTs for cued and uncued trials as a function of cue-target stimulus onset asynchrony with standard error bars.

Figure 6. Attentional Blink: Mean T2 Proportion Correct as a function of T1–T2 Lag.

Mean T2 (second target) proportion correct as a function of T1–T2 lag with standard error bars.

Figure 7. Masked Priming: Compatible and Incompatible Mean RTs and Error Rates Across Prime Durations.

Mean RTs and error rates for compatible and incompatible masked prime trials as a function of prime duration with standard error bars.

Figure 8. Cognitive Learning: A comparison between the learning curves reported in Nosfosky et al. (1994) data and the AMT replication data in Experiment 8.

The probability of classification error as a function of training block. The top panel shows the learning curves estimated by Nosfosky et al. using 120 participants (40 per learning problem) who each performed two randomly selected problems. The right panel shows our AMT data with 228 participants, each who performed only one problem (38 per condition). We ended the experiment after 15 blocks, although Nosofsky et al. stopped after 25. Thus, the Nosofsky et al. data have been truncated to facilitate visual comparison.

Figure 9. Cognitive Learning: The average number of block to criterion for each problem, an index of problem difficulty.

The average number of blocks it took participants to reach criterion (2 blocks of 16 trials in a row with no mistakes) in each problem. The white bars show the estimated average number of blocks to criterion reported by Nosofsky et al. .

Figure 10. Cognitive Learning: The learning curves for Shepard et al. Type II and IV problems based on task incentives.

The probability of classification error as a function of training block, learning problem and incentive for Experiment 9. The incentive structure had little impact on performance within each problem.

Figure 11. Cognitive Learning: The learning curves for Shepard et al. problems I, II, IV, and VI in Experiment 10.

The top panel compared the results of Nosfosky et al. to the results of Experiment 10. The bottom compares the results of Lewandowsky to the results of Experiment 10 giving two different views of the relationship between the online and laboratory based data. Overall, the Type II problem seems more difficult than in previous report (as is the Type VI). However, in general, the instruction manipulation increased the congruence between the online and laboratory data.