Methodological considerations for behavioral studies relying on response time outcomes through online crowdsourcing platforms

Abstract

This perspective paper explores challenges associated with online crowdsourced data collection, particularly focusing on longitudinal tasks with time-sensitive outcomes like response latencies. Based on our research, we identify two significant sources of bias: technical shortcomings such as low, variable frame rates, and human factors, contributing to high attrition rates. We explored potential solutions to these problems, such as enforcing hardware acceleration and defining study-specific frame rate thresholds, as well as pre-screening participants and monitoring hardware performance and task engagement over each experimental session. With this discussion, we intend to provide recommendations on how to improve the quality and reliability of data collected via online crowdsourced platforms and emphasize the need for researchers to be cognizant of potential pitfalls in online research.

Figure 1

Hardware acceleration influences on frame rate and frame rate variability. Hardware acceleration disabled (‘off’) shown in black, enabled (‘on’) depicted in red. (a) Comparison of Pavlovia and custom frame rate calculations. Frames per second (FPS) shown for both frame rate sources, i.e., Pavlovia’s built-in (left) that sampled one data point per session onset, and custom (right), which sampled frame rate across trials within session. (b) Normalized distributions of frame rate. Depicts effects of hardware acceleration on custom frame rate calculation across all available data with hardware acceleration ‘on’ showing higher and less variable frame rates than ‘off’. (c) Variability distributions of custom frame rate. Depicts within session, between-trial variability (standard deviation, std) of custom frame rate for individual workers with hardware acceleration ‘on’ showing less variability within session than ‘off’. Data and code to reproduce the figures are available at 10.6084/m9.figshare.23499195.

Figure 2

Sources of attrition and noncompliance over time. (a) Worker attrition over 7 consecutive days. Colors reflect data loss due to rejection (gray, top), loss to follow-up (orange, middle), and data retained (blue, bottom) over the seven-day experiment. (b) Flowchart of attrition from screening to final sample. Diminishing circle sizes reflect the loss of data from screening through session 1 and session 7. Circle colors reflect data loss due to noncompliance (orange, right), data retention (blue, left), and final sample (dark blue, bottom left). Numbers within circles reflect worker count. (c) Geolocation of workers across 2277 sessions. Colors represent whether the country from which data originated was on the list targeted for recruitment (dark blue) or not (red). Data and code to reproduce (a) and (c) are available at 10.6084/m9.figshare.23499198.