Explaining variation in the effects of welfare-to-work programs

Abstract

Evaluations of government-funded training programs often combine results from similar operations in multiple sites. Findings inevitably vary. It is common to relate site-to-site variations in outcomes to variations in program design, participant characteristics, and the local environment. Frequently, such connections are constructed in a narrative synthesis of multisite results. This article uses findings from the evaluations of California's Greater Avenues for Independence (GAIN) program and the National Evaluation of Welfare-to-Work Strategies (NEWWS) to illustrate why it is important to question the legitimacy of such syntheses. The discussion is carried out using a simple multilevel evaluation model that incorporates models of both individual outcomes within sites and variation in program effects across sites. The results indicate that tempting generalizations about GAIN and NEWWS effects are statistically unjustified but that significant progress might be made in identifying the determinants of program effects in future demonstrations with some changes in evaluation strategy.