Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2022 Nov 2;11(5):1089-1109.
doi: 10.1093/jssam/smac028. eCollection 2023 Nov.

Estimating Web Survey Mode and Panel Effects in a Nationwide Survey of Alcohol Use

Randal ZuWallack  1 , Matt Jans  2 , Thomas Brassell  3 , Kisha Bailly  4 , James Dayton  5 , Priscilla Martinez  6 , Deidre Patterson  7 , Thomas K Greenfield  8 , Katherine J Karriker-Jaffe  9
Affiliations
Review

Estimating Web Survey Mode and Panel Effects in a Nationwide Survey of Alcohol Use

Randal ZuWallack et al. J Surv Stat Methodol. .

Abstract

Random-digit dialing (RDD) telephone surveys are challenged by declining response rates and increasing costs. Many surveys that were traditionally conducted via telephone are seeking cost-effective alternatives, such as address-based sampling (ABS) with self-administered web or mail questionnaires. At a fraction of the cost of both telephone and ABS surveys, opt-in web panels are an attractive alternative. The 2019-2020 National Alcohol Survey (NAS) employed three methods: (1) an RDD telephone survey (traditional NAS method); (2) an ABS push-to-web survey; and (3) an opt-in web panel. The study reported here evaluated differences in the three data-collection methods, which we will refer to as "mode effects," on alcohol consumption and health topics. To evaluate mode effects, multivariate regression models were developed predicting these characteristics, and the presence of a mode effect on each outcome was determined by the significance of the three-level effect (RDD-telephone, ABS-web, opt-in web panel) in each model. Those results were then used to adjust for mode effects and produce a "telephone-equivalent" estimate for the ABS and panel data sources. The study found that ABS-web and RDD were similar for most estimates but exhibited differences for sensitive questions including getting drunk and experiencing depression. The opt-in web panel exhibited more differences between it and the other two survey modes. One notable example is the reporting of drinking alcohol at least 3-4 times per week, which was 21 percent for RDD-phone, 24 percent for ABS-web, and 34 percent for opt-in web panel. The regression model adjusts for mode effects, improving comparability with past surveys conducted by telephone; however, the models result in higher variance of the estimates. This method of adjusting for mode effects has broad applications to mode and sample transitions throughout the survey research industry.

Keywords: Alcohol survey; Mode transition; Multi-mode; Nonprobability.

PubMed Disclaimer

References

    1. Baker R., Brick J. M., Bates N. A., Battaglia M., Couper M. P., Dever J. A., Gile K. J., Tourangeau R. (2013), “Summary Report of the AAPOR Task Force on Non-Probability Sampling,” Journal of Survey Statistics and Methodology, 1, 90–143.
    1. Beck F., Guignard R., Legleye S. (2014), “Does Computer Survey Technology Improve Reports on Alcohol and Illicit Drug Use in the General Population? A Comparison between Two Surveys with Different Data Collection Modes in France,” PLoS One, 9, e85810. - PMC - PubMed
    1. Boyle J., Berman L., Dayton J., Iachan R., Jans M., Locke R., ZuWallack R. (2019), “Topic Salience and Propensity to Respond to Surveys: Findings from a National Mobile Panel.” April 17. Available at: https://www.census.gov/fedcasic/fc2019/ppt/7ABoyleTopicSalience.pdf.
    1. Brosnan K., Kemperman A., Dolnicar S. (2021), “Maximizing Participation from Online Survey Panel Members,” International Journal of Market Research, 63, 416–435.
    1. Cornesse C., Blom A. G., Dutwin D., Krosnick J. A., De Leeuw E. D., Legleye S., Pasek J., Pennay D., Phillips B., Sakshaug J. W., Struminskaya B., Wenz A. (2020), “A Review of Conceptual Approaches and Empirical Evidence on Probability and Nonprobability Sample Survey Research,” Journal of Survey Statistics and Methodology, 8, 4–36.