Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Aug;30(4):409-414.
doi: 10.1037/pha0000549. Epub 2022 Feb 17.

Are poor quality data just random responses?: A crowdsourced study of delay discounting in alcohol use disorder

William H Craft  1 , Allison N Tegge  1 , Roberta Freitas-Lemos  1 , Devin C Tomlinson  1 , Warren K Bickel  1
Affiliations

Are poor quality data just random responses?: A crowdsourced study of delay discounting in alcohol use disorder

William H Craft et al. Exp Clin Psychopharmacol. 2022 Aug.

Abstract

Crowdsourced methods of data collection such as Amazon Mechanical Turk (MTurk) have been widely adopted in addiction science. Recent reports suggest an increase in poor quality data on MTurk, posing a challenge to the validity of findings. However, empirical investigations of data quality in addiction-related samples are lacking. In this study of individuals with alcohol use disorder (AUD), we compared poor quality delay discounting data to randomly generated data. A reanalysis of prior published delay discounting data was conducted comparing included, excluded, and randomly generated data samples. Nonsystematic criteria were implemented as a measure of data quality. The excluded data was statistically different from the included sample but did not differ from randomly generated data on multiple metrics. Moreover, a response bias was identified in the excluded data. This study provides empirical evidence that poor quality delay discounting data in an AUD sample is not statistically different from randomly generated data, suggesting data quality concerns on MTurk persist in addiction samples. These findings support the use of rigorous methods of a priori defined criteria to remove poor quality data post hoc. Additionally, it highlights that the use of nonsystematic delay discounting criteria to remove poor quality data is rigorous and not simply a way of removing data that does not conform to an expected theoretical model. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

PubMed Disclaimer

Figures

Figure 1.
Figure 1.
Summary of discounting data for the total, included, excluded, and randomly generated samples. Mean (±SEM) discounting curves (left panel) and area under the curve values (right panel) in the active and control narrative conditions for the (A) total, (B) included, (C) excluded, and (D) randomly generated samples. SEM: standard error of the mean. Note; ns indicates non-significant relationship, ** indicates p-value = 0.001.
See this image and copyright information in PMC

References

    1. Bailey AJ, Romeu RJ, & Finn PR (2021). The problems with delay discounting: a critical review of current practices and clinical applications. Psychological Medicine, 1–8. - PMC - PubMed
    1. Chmielewski M, & Kucker SC (2020). An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results. Social Psychological and Personality Science, 11(4), 464–473.
    1. Craft WH, Tegge AN, & Bickel WK (2021). Narrative theory IV: Within-subject effects of active and control scarcity narratives on delay discounting in alcohol use disorder. Experimental and Clinical Psychopharmacology. 10.1037/pha0000478 - DOI - PubMed
    1. Dreyfuss E (2018, August 17). A Bot Panic Hits Amazon’s Mechanical Turk. Wired https://www.wired.com/story/amazon-mechanical-turk-bot-panic/
    1. Du W, Green L, & Myerson J (2002). Cross-Cultural Comparisons of Discounting Delayed and Probabilistic Rewards. The Psychological Record, 52(4), 479–492.