Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2013 Nov 29;15(11):e260.
doi: 10.2196/jmir.2903.

Comparison of US panel vendors for online surveys

Benjamin M Craig  1 , Ron D Hays, A Simon Pickard, David Cella, Dennis A Revicki, Bryce B Reeve
Affiliations

Comparison of US panel vendors for online surveys

Benjamin M Craig et al. J Med Internet Res. .

Abstract

Background: Despite the increasing use of panel surveys, little is known about the differences in data quality across panels.

Objective: The aim of this study was to characterize panel survey companies and their respondents based on (1) the timeliness of response by panelists, (2) the reliability of the demographic information they self-report, and (3) the generalizability of the characteristics of panelists to the US general population. A secondary objective was to highlight several issues to consider when selecting a panel vendor.

Methods: We recruited a sample of US adults from 7 panel vendors using identical quotas and online surveys. All vendors met prespecified inclusion criteria. Panels were compared on the basis of how long the respondents took to complete the survey from time of initial invitation. To validate respondent identity, this study examined the proportion of consented respondents who failed to meet the technical criteria, failed to complete the screener questions, and provided discordant responses. Finally, characteristics of the respondents were compared to US census data and to the characteristics of other panels.

Results: Across the 7 panel vendors, 2% to 9% of panelists responded within 2 days of invitation; however, approximately 20% of the respondents failed the screener, largely because of the discordance between self-reported birth date and the birth date in panel entry data. Although geographic characteristics largely agreed with US Census estimates, each sample underrepresented adults who did not graduate from high school and/or had annual incomes less than US $15,000. Except for 1 vendor, panel vendor samples overlapped one another by approximately 20% (ie, 1 in 5 respondents participated through 2 or more panel vendors).

Conclusions: The results of this head-to-head comparison provide potential benchmarks in panel quality. The issues to consider when selecting panel vendors include responsiveness, failure to maintain sociodemographic diversity and validated data, and potential overlap between panels.

Keywords: Internet; community surveys; data sources; sampling bias; selection bias; survey methods.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: None declared.

Figures

Figure 1
Figure 1
Proportion of invited panels who consented by day and panel vendor.
Figure 2
Figure 2
Proportion of consented respondents who failed the screener by reason and panel vendor.
Figure 3
Figure 3
Proportion of discordant responses by self-reported attribute and panel vendor.
Figure 4
Figure 4
Weighted proportion of respondents who passed screener by educational attainment and panel vendor (PV).
Figure 5
Figure 5
Weighted proportion of respondents who passed screener by annual household income in 2011 and panel vendor (PV).
Figure 6
Figure 6
Weighted proportion of respondents who passed screener by US Census division and panel vendor (PV).
See this image and copyright information in PMC

References

    1. Baker R, Blumberg S, Brick JM, Couper MP, Courtright M, Dennis M, Dillman D, Frankel MR, Garland P, Groves RM, Kennedy C, Krosnick J, Lee S, Lavrakas P, Link M, Piekarski L, Rao K, Rivers D, Thomas RK, Zahs D. AAPOR Report on Online Panels. 2010. Mar, [2013-11-12]. http://www.aapor.org/AM/Template.cfm?Section=AAPOR_Committee_and_Task_Fo....
    1. Berrens RP, Bohara AK, Jenkins-Smith H, Silva C, Weimer DL. The advent of Internet surveys for political research: A comparison of telephone and Internet samples. Polit Anal. 2003;11:1–22. doi: 10.1093/pan/11.1.1. - DOI
    1. Duffy B, Smith K, Terhanian G, Bremer J. Comparing data from online and face-to-face surveys. Int J Market Res. 2005;47:615–639.
    1. Grandjean BD, Taylor PA, Nelson NM. Comparing an Internet panel survey to mail and phone surveys on “willingness to pay” for environmental quality: a national mode test. The American Association for Public Opinion Research (AAPOR) 64th Annual Conference, May; The American Association for Public Opinion Research (AAPOR) 64th Annual Conference; May 14-17, 2009; Hollywood, FL. Hollywood, FL: 2009. pp. 5779–5793.
    1. Hirsch O, Hauschild F, Schmidt MH, Baum E, Christiansen H. Comparison of Web-based and paper-based administration of ADHD questionnaires for adults. J Med Internet Res. 2013;15(3):e47. doi: 10.2196/jmir.2225. http://www.jmir.org/2013/3/e47/ - DOI - PMC - PubMed

LinkOut - more resources