Opening the black box: using process evaluation measures to assess implementation and theory building

Abstract

The past decade has seen increasing recognition in prevention science of the need to move away from a black box approach to intervention evaluation and toward an approach that can elaborate on the mechanisms through which changes in the outcomes operate (Chen & Rossi, 1989; Durlak & Wells, 1997; Spoth et al., 1995). An approach that examines issues of program implementation is particularly critical in the design of efficacy studies of school-based preventive interventions. Numerous preventive intervention strategies are now delivered within the schools, often by regular classroom teachers. The extent to which teachers faithfully deliver a particular curriculum or incorporate instructional strategies emphasized by an intervention is a critical question for the overall project evaluation. This article illustrates the utilization of process measures from a multicomponent school-based prevention program to examine implementation of a teaching staff development intervention, and the program's underlying theoretical basis. Given the nested study design, the analyses utilize hierarchical linear models (Bryk & Raudenbush, 1992) to examine changes in teaching strategies by condition and investigate the hypothesized relationships between teaching practices and student behaviors based on the program's theoretical framework. Results suggest that teaching practices in two of the six intervention focus areas were positively impacted in the first 18 months of the project. Findings also support the relationships between teachers' instructional practices and students' behavior.