Does participation in an intervention affect responses on self-report questionnaires?

Abstract

There has been some concern that participation in an intervention and exposure to a measurement instrument can change participants' interpretation of the items on a self-report questionnaire thereby distorting subsequent responses and biasing results. Differential item functioning (DIF) analysis using item response modeling can ascertain possible differences in item interpretation by testing for differences in item location between groups. The DIF for treatment versus control group differences at post-intervention assessment and the Time 1 and Time 2 differences in a control group were analyzed using data from a dietary change intervention trial for Boy Scouts. The measures included fruit and vegetable (FV) frequency of consumption, preferences and self-efficacy. Treatment-control group DIF at post-intervention assessment was detected in a higher percentage of items for FV frequency than for preference or self-efficacy. Time 1 to Time 2 differences in items for the control group were detected in one item for each of the three scales. Further research will need to clarify whether the obtained DIFs reflected true changes in frequency, preference or self-efficacy or some reinterpretation of items by participants following an intervention or merely after previous exposure to the measure.