Protecting the integrity of survey research

Kathleen Hall Jamieson¹, Arthur Lupia², Ashley Amaya³, Henry E Brady⁴, René Bautista⁵, Joshua D Clinton⁶, Jill A Dever⁷, David Dutwin⁸, Daniel L Goroff⁹, D Sunshine Hillygus¹⁰, Courtney Kennedy³, Gary Langer¹¹, John S Lapinski¹², Michael Link¹³, Tasha Philpot¹⁴, Ken Prewitt¹⁵, Doug Rivers¹⁶, Lynn Vavreck¹⁷, David C Wilson¹⁸, Marcia K McNutt¹⁹

Affiliations

¹ Annenberg Public Policy Center, University of Pennsylvania, Philadelphia, PA 19104, USA.
² Department of Political Science, University of Michigan, 505 South State Street, Ann Arbor, MI 48106, USA.
³ Pew Research Center, 1615 L Street NW #800, Washington, DC 20036, USA.
⁴ Goldman School of Public Policy, University of California, 2607 Hearst Ave, Berkeley, CA 94720, USA.
⁵ Methodology and Quantitative Social Science Department, NORC at the University of Chicago, Chicago, IL 60637, USA.
⁶ Department of Political Science, Vanderbilt University, 230 Appleton Place, Nashville, TN 37203, USA.
⁷ Division for Statistical and Data Sciences, RTI International, Washington, DC 20005-3967, USA.
⁸ Center for Panel Survey Sciences, NORC at the University of Chicago, 55 East Monroe Street, Chicago, IL 60618, USA.
⁹ Alfred P. Sloan Foundation, 630 Fifth Avenue, New York, NY 10111, USA.
¹⁰ Department of Political Science, Duke University, 140 Science Dr, Durham, NC 27707, USA.
¹¹ Langer Research Associates, Carmel, NY 10512, USA.
¹² Department of Political Science, University of Pennsylvania, 133 South 36th Street, Philadelphia, PA 19104, USA.
¹³ Ipsos, Canton, GA 30115, USA.
¹⁴ Department of Government, University of Texas at Austin, 158 W 21st Street, Austin, TX 78712, USA.
¹⁵ Department of Political Science, Columbia University, New York City, NY 10012, USA.
¹⁶ Department of Political Science, Stanford University, Stanford, CA 94305, USA.
¹⁷ Department of Political Science, University of California Los Angeles, Los Angeles, CA 90095, USA.
¹⁸ Goldman School of Public Policy, University of California, Berkeley, Berkeley, CA 94720, USA.
¹⁹ National Academy of Sciences, 500 5th Street NW, Washington, DC 20001, USA.

PMID: 36999096
PMCID: PMC10047603
DOI: 10.1093/pnasnexus/pgad049

Protecting the integrity of survey research

Kathleen Hall Jamieson et al. PNAS Nexus. 2023.

. 2023 Mar 28;2(3):pgad049.

doi: 10.1093/pnasnexus/pgad049. eCollection 2023 Mar.

Authors

Affiliations

¹ Annenberg Public Policy Center, University of Pennsylvania, Philadelphia, PA 19104, USA.
² Department of Political Science, University of Michigan, 505 South State Street, Ann Arbor, MI 48106, USA.
³ Pew Research Center, 1615 L Street NW #800, Washington, DC 20036, USA.
⁴ Goldman School of Public Policy, University of California, 2607 Hearst Ave, Berkeley, CA 94720, USA.
⁵ Methodology and Quantitative Social Science Department, NORC at the University of Chicago, Chicago, IL 60637, USA.
⁶ Department of Political Science, Vanderbilt University, 230 Appleton Place, Nashville, TN 37203, USA.
⁷ Division for Statistical and Data Sciences, RTI International, Washington, DC 20005-3967, USA.
⁸ Center for Panel Survey Sciences, NORC at the University of Chicago, 55 East Monroe Street, Chicago, IL 60618, USA.
⁹ Alfred P. Sloan Foundation, 630 Fifth Avenue, New York, NY 10111, USA.
¹⁰ Department of Political Science, Duke University, 140 Science Dr, Durham, NC 27707, USA.
¹¹ Langer Research Associates, Carmel, NY 10512, USA.
¹² Department of Political Science, University of Pennsylvania, 133 South 36th Street, Philadelphia, PA 19104, USA.
¹³ Ipsos, Canton, GA 30115, USA.
¹⁴ Department of Government, University of Texas at Austin, 158 W 21st Street, Austin, TX 78712, USA.
¹⁵ Department of Political Science, Columbia University, New York City, NY 10012, USA.
¹⁶ Department of Political Science, Stanford University, Stanford, CA 94305, USA.
¹⁷ Department of Political Science, University of California Los Angeles, Los Angeles, CA 90095, USA.
¹⁸ Goldman School of Public Policy, University of California, Berkeley, Berkeley, CA 94720, USA.
¹⁹ National Academy of Sciences, 500 5th Street NW, Washington, DC 20001, USA.

PMID: 36999096
PMCID: PMC10047603
DOI: 10.1093/pnasnexus/pgad049

Abstract

Although polling is not irredeemably broken, changes in technology and society create challenges that, if not addressed well, can threaten the quality of election polls and other important surveys on topics such as the economy. This essay describes some of these challenges and recommends remediations to protect the integrity of all kinds of survey research, including election polls. These 12 recommendations specify ways that survey researchers, and those who use polls and other public-oriented surveys, can increase the accuracy and trustworthiness of their data and analyses. Many of these recommendations align practice with the scientific norms of transparency, clarity, and self-correction. The transparency recommendations focus on improving disclosure of factors that affect the nature and quality of survey data. The clarity recommendations call for more precise use of terms such as "representative sample" and clear description of survey attributes that can affect accuracy. The recommendation about correcting the record urges the creation of a publicly available, professionally curated archive of identified technical problems and their remedies. The paper also calls for development of better benchmarks and for additional research on the effects of panel conditioning. Finally, the authors suggest ways to help people who want to use or learn from survey research understand the strengths and limitations of surveys and distinguish legitimate and problematic uses of these methods.

Keywords: scientific integrity; survey research; weighting of data.

PubMed Disclaimer

References

1. Visser PS, Krosnick JA, Lavrakas PJ. 2000. Survey research. In: Reis HT, Judd CM, editors. Handbook of research methods in social and personality psychology. New York (NY): Cambridge University Press. p. 223–252.
1. Santos R. 2014. Presidential address: borne of a renaissance–a metamorphosis for our future. Public Opin Quart. 78(3):769–777.
1. Link M. 2015. Presidential address: AAPOR2025 and the opportunities in the decade before US. Public Opin Quart. 79(3):828–836.
1. Pew Research Center . 2014. Political polarization in the American public. Pew Research Center. https://www.pewresearch.org/politics/2014/06/12/political-polarization-i....
1. Rainie L, Perrin A. 2019, July 22. Key findings about Americans’ declining trust in government and each other. Pew Research Center. https://www.pewresearch.org/fact-tank/2019/07/22/key-findings-about-amer....

LinkOut - more resources

Full Text Sources
- eScholarship, University of California - Access Free Full Text

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Protecting the integrity of survey research

Affiliations

Protecting the integrity of survey research

Authors

Affiliations

Abstract

References

LinkOut - more resources

Full Text Sources