Comparing attentional disengagement between Prolific and MTurk samples

Abstract

Attention often disengages from primary tasks in favor of secondary tasks (i.e., multitasking) and task-unrelated thoughts (i.e., mind wandering). We assessed whether attentional disengagement, in the context of a cognitive task, can substantially differ between samples from commonly used online participant recruitment platforms, Prolific and Mechanical Turk (MTurk). Initially, eighty participants were recruited through Prolific to perform an attention task in which the risk of losing points for errors was varied (high risk = 80% chance of loss, low risk = 20% chance of loss). Attentional disengagement was measured via task performance along with self-reported mind wandering and multitasking. On Prolific, we observed surprisingly low levels of disengagement. We then conducted the same experiment on MTurk. Strikingly, MTurk participants exhibited more disengagement than Prolific participants. There was also an interaction between risk and platform, with the high-risk group exhibiting less disengagement, in terms of better task performance, than the low-risk group, but only on MTurk. Platform differences in individual traits related to disengagement and relations among study variables were also observed. Platform differences persisted, but were smaller, after increasing MTurk reputation criteria and remuneration in a second experiment. Therefore, recruitment platform and recruitment criteria could impact results related to attentional disengagement.

Figure 1

Attention task with performance feedback and risk manipulation. There were two to six non-target displays before a target display. There was a 20% chance of a mind wandering probe preceding performance feedback. The “x” on the line between the probe and bottom right performance feedback displays represents the chance of participants losing points for an incorrect response. The “100%-x” on the line between the probe and bottom left performance feedback displays represents the chance of not losing points for an incorrect response.

Figure 2

Flow of participants through E1 and E2 by platform. Participants were randomized upon loading the task instructions screen. Participants that reported an age greater than 35 years in the demographics questionnaire were excluded from analyses.

Figure 3

Accuracy, mind wandering, and risk perception by risk condition, recruitment platform, and experiment. Accuracy reflects the percentage of targets to which participants responded correctly. Mind wandering reflects the percentage of thought probes to which participants indicated off-task thoughts. Risk perception reflects participant estimates of the chance that task errors led to point loss. Actual chances were: 20% in the low-risk condition; and 80% in the high-risk condition. Points reflect raw means and error bars reflect 95% confidence intervals.

Figure 4

Self-reported rate and impact of multitasking by recruitment platform and experiment. Multitasking was assessed by asking participants, “While completing this study, were you engaged in any media-related activities outside of the contents of the experiment?” Response options included: (1) “Yes”; (2) “No, I was not engaged in any activities outside of the contents of this study”; and (3) “No, but I was engaged in other, media-unrelated activities while completing this study.” If participants selected options 1 (media-related multitasking) or 3 (media-unrelated multitasking), they were asked to what extent the other activities: (a) consumed attention; and (b) affected performance. Participants were assured that they would not be penalized for responding one way or another.

Figure 5

Correlations between individual traits, task performance, mind wandering, and risk perception for Prolific (upper triangles) and MTurk (lower triangles) by experiment. Acc = accuracy (percentage of correct responses to targets), ARCES = Attention-Related Cognitive Errors Scale, BSSS = Brief Sensation Seeking Scale, MAAS = Mindful Awareness of Attention Scale, MW = mind wandering (percentage of off-task reports to thought probes), WCFS = Workplace Cognitive Failures Scale, Risk = risk perception (perception of the risk of losing points for task errors). Results are collapsed across low-risk and high-risk conditions. Coefficients were generated with Spearman’s rank-order method. *p < .05, **p < .01, ***p < .001.