Methodological problems in the retrospective computation of responsiveness to change: the lesson of Cronbach

Abstract

Objective: To examine the relation between responsiveness coefficients derived directly from a calculation of average change resulting from a treatment intervention (Responsiveness-Treatment or RT) and those derived from retrospective analysis of changed and unchanged groups (Responsiveness Retrospective or RR) based on a global measure of change.

Method: Two approaches were used. First, we used simulation methods to examine the analytical relationship between the RT and RR coefficients. We then located eight studies where it was possible to compute both RT and RR coefficients. As anticipated from theoretical arguments, the RR coefficients were larger than the RT coefficients (1.50 versus 0.41, p < .0001). Within study there was no predictable relationship between the two indices. Across studies, the magnitude of the RR coefficient was strongly related to the correlation with the retrospective global scale, and unrelated to the magnitude of the RT coefficient. The simulated curves fit well with the observed data, and substantiated the observation that the relation between RT and RR coefficients is complex and only weakly related to the size of the treatment effect.

Conclusion: Retrospective methods of computing responsiveness yield little information about the ability of an instrument to detect treatment effects, and should not be used as a basis for choice of an instrument for applications to clinical trials.