Avoiding type III errors in health education program evaluations: a case study

Abstract

Monitoring the implementation of a program being evaluated can improve the interpretability of data collected and help evaluators to avoid committing a Type III error: evaluating a program that has not been adequately implemented. This article describes an evaluation that analyzed the implementation of a school health education curriculum, assessed cognitive learning outcomes attributable to the curriculum, and examined the relationship between classroom implementation and changes in students' knowledge. Five fifth-grade classes (n = 101) participated in the curriculum, and five classes (n = 84) served as a comparison group. Data collection procedures involved a pretest and posttest of all students' health-related knowledge, daily monitoring of classroom implementation by the five teachers participating, and questionnaires completed by principals and teachers. Analysis methods included descriptive statistics, parametric and nonparametric tests of significance, and qualitative assessment procedures. Results indicated that the curriculum had a positive effect on learning in students; curriculum implementation varied considerably among the five classes participating; teaching/learning activities that were most and least likely to be implemented could be identified and described; both teachers and principals perceived the program favorably; some health instruction was occurring in the comparison classes, so it was not appropriate to consider them as pure controls; and no statistically significant relationship between curriculum implementation and cognitive outcomes was observed. This study provides evidence of the need for and value of measuring implementation of programs being evaluated. Implications for developing implementation measures and the role of formative evaluation in health education practice are considered.