Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2013 Winter;32(1):107-121.
doi: 10.1002/pam.21660.

External Validity in Policy Evaluations that Choose Sites Purposively

Robert B Olsen  1 , Larry L Orr  2 , Stephen H Bell  1 , Elizabeth A Stuart  3
Affiliations

External Validity in Policy Evaluations that Choose Sites Purposively

Robert B Olsen et al. J Policy Anal Manage. 2013 Winter.

Abstract

Evaluations of the impact of social programs are often carried out in multiple "sites," such as school districts, housing authorities, local TANF offices, or One-Stop Career Centers. Most evaluations select sites purposively following a process that is nonrandom. Unfortunately, purposive site selection can produce a sample of sites that is not representative of the population of interest for the program. In this paper, we propose a conceptual model of purposive site selection. We begin with the proposition that a purposive sample of sites can usefully be conceptualized as a random sample of sites from some well-defined population, for which the sampling probabilities are unknown and vary across sites. This proposition allows us to derive a formal, yet intuitive, mathematical expression for the bias in the pooled impact estimate when sites are selected purposively. This formula helps us to better understand the consequences of selecting sites purposively, and the factors that contribute to the bias. Additional research is needed to obtain evidence on how large the bias tends to be in actual studies that select sites purposively, and to develop methods to increase the external validity of these studies.

Keywords: Evaluation; convenience sampling; external validity; generalizability; impacts; purposive sampling; random assignment.

PubMed Disclaimer

References

    1. Bloom HS, Michalopoulos C, Hill CJ. Using experiments to assess nonexperimental comparison-group methods for measuring program effects. In: Bloom HS, editor. Learning more from social experiments. New York: Russell Sage Foundation; 2005. pp. 173–235.
    1. Bos JM, Fellerath V. Final report on Ohio’s welfare initiative to improve school attendance among teenage parents: Ohio’s Learning, Earning, and Parenting Program. New York: Manpower Demonstrative Research Corporation; 1997.
    1. Brick JM, Jones ME. Propensity to respond and nonresponse bias. Metron—International Journal of Statistics. 2008;6:51–73.
    1. Cole SR, Stuart EA. Generalizing evidence from randomized clinical trials to target populations: the ACTG-320 trial. American Journal of Epidemiology. 2010;172:107–115. Available at http://aje.oxfordjournals.org/cgi/content/abstract/172/1/107?etoc. - PMC - PubMed
    1. Cook TD, Shadish WR, Wong VC. Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management. 2008;27:724–750.

LinkOut - more resources