Population Recruitment Strategies in the Age of Bots: Insights from the What Is on Your Plate Study

Abstract

Background: To evaluate state-wide nutrition policies, valid tools are required to gather sufficient sample sizes. Remote data collection, including web-based dietary assessments, offers convenience for participants and researchers and enables faster and more diverse recruitment. However, it presents challenges, including risk of bots compromising data integrity.

Objectives: This study describes the technical survey design of an ongoing longitudinal study, which is evaluating a state-wide Supplemental Nutrition Assistance Program (SNAP) incentive program, discusses strategies to prevent and identify bots, duplicates, fraudulent entries, and implausible data, and provides recommendations to improve future public health nutrition research.

Methods: From May to September 2023, SNAP participants from Rhode Island and Connecticut were recruited to complete an online food frequency questionnaire (FFQ) and a demographic survey. Given the large sample and online format, our interdisciplinary team designed the technical backend to optimize participants' convenience while ensuring data quality through an automated system that assessed FFQ responses. To prevent bots and duplicates, we created duplicate application programming interfaces (API), randomly called participants, and evaluated Completely Automated Public Turing Test to Tell Computers and Humans Apart (reCAPTCHA), geotags, and Internet Protocol (IP) addresses.

Results: Using a combination of text blasts and in-person recruitment, we enrolled 1367 participants, with text blasts proving the most effective strategy (∼60% of participants). Midway through recruitment, we identified 544 potential bots that completed the screener, with duplicate IP addresses and geotags from outside the recruitment area serving as strong indicators of bot activity. At baseline, 112 participants failed FFQ data quality checks, prompting follow-up by research assistants. Our automated duplicate and FFQ APIs saved countless hours of staff time.

Conclusions: Remote data collection tools were critical for meeting recruitment goals and ensuring our data authenticity. A combination of strategies is necessary to effectively mitigate against bots and ensure plausible responses. Widely available, built-in tools (e.g., reCAPTCHA) are helpful but are insufficient alone. Customized solutions like our automated systems may be critical for future researchers to maintain data integrity.

Keywords: Supplemental Nutrition Assistance Program; bots; data integrity; dietary assessment; fraud prevention; online survey security; web-based food frequency questionnaire.

FIGURE 1

Baseline survey structure. ¹Duplicate flag as detected by Qualtrics or duplicate email or phone detected by the API. ²Flag triggered if age provided in screener did not match date of birth in full survey and/or if participant completed food frequency questionnaire in <10 min, or if participant reported < 600 kcals or > 10,000 kcal. APIs, application programming interfaces; AWS, Amazon Web Services; DOB, date of birth; FFQ, food frequency questionnaire; QA, quality assurance; QR, quick response.

FIGURE 2

Follow-up survey structure. ¹Receipt flag if receipt does not show EBT card was used, receipt older than 30 days from submission date, or receipt is a duplicate (duplicate photo or EBT card number). ²QA flag triggered if age provided in screener did not match date of birth in full survey and/or if participant completed food frequency questionnaire in < 10 min, or if participant reported < 600 kcals or > 10,000 kcal. APIs, application programming interfaces; EBT, electronic benefit transfer; FFQ, food frequency questionnaire; QA, quality assurance.

FIGURE 3

Quality assurance data check process and structure. APIs, application programming interfaces; FFQ, food frequency questionnaire; QA, quality assurance.