Methods for improving participation rates in national self-administered web/mail surveys: Evidence from the United States

Abstract

In the United States, increasing access to the internet, the increasing costs of large-scale face-to-face data collections, and the general reluctance of the public to participate in intrusive in-person data collections all mean that new approaches to nationally representative surveys are urgently needed. The COVID-19 pandemic accelerated the need for faster, higher-quality alternatives to face-to-face data collection. These trends place a high priority on the evaluation of innovative web-based data collection methods that are convenient for the U.S. public and yield scientific information of high quality. The web mode is particularly appealing because it is relatively inexpensive, it is logistically flexible to implement, and it affords a high level of privacy and confidentiality when correctly implemented. With this study, we aimed to conduct a methodological evaluation of a sequential mixed-mode web/mail data collection protocol, including modular survey design concepts, which was implemented on a national probability sample in the U.S. in 2020-2021. We implemented randomized experiments to test theoretically-informed hypotheses that 1) the use of mail and increased incentives to follow up with households that did not respond to an invitation to complete a household screening questionnaire online would help to recruit different types of households; and 2) the use of modular survey design, which involves splitting a lengthy self-administered survey up into multiple parts that can be completed at a respondent's convenience, would improve survey completion rates. We find support for the use of mail and increased incentives to follow up with households that have not responded to a web-based screening questionnaire. We did not find support for the use of modular design in this context. Simple descriptive analyses also suggest that attempted telephone reminders may be helpful for the main survey.

Fig 1. Screening and main data collection protocols for the AFHS.

Fig 2. Phase-specific response rates at the screening stage.

Fig 3. Cumulative phase-specific response rates at the screening stage, in addition to hypothetical response rates at the conclusion of Phase 4 if all cases had either received or not received the $5 + priority mailing.