Methodological issues in cross-syndrome comparisons: matching procedures, sensitivity (Se), and specificity (Sp)

Abstract

We are impressed with the magnitude and potential importance of the studies presented by Sigman and Ruskin in this monograph. The within-syndrome findings for the children with autism concerning relations between early joint attention and a range of cognitive abilities a full 9 years later provide the strongest evidence so far that early nonverbal communication skills play an important role in the later development of language, intelligence, and social relations with peers. The purpose of the monograph was not limited to within-syndrome research questions, however. Sigman and Ruskin state that a major goal of the research reported in the monograph was to identify specific, unique, and universal deficits for autism and Down syndrome. They base their method of identifying such syndrome characteristics on the group-matching procedure. Given that this procedure is fraught with difficulties, we are concerned that many of Sigman and Ruskin's cross-syndrome comparisons may be incorrect. We do not mean to single out Sigman and Ruskin. The group-matching method is frequently used in special populations research, with the null hypothesis of no differences on the control variable being accepted at dangerously low p values. Our concerns with the group-matching problem extend to much of the extant research that attempts to identify characteristics of individuals based on the performance of their syndrome group relative to a control group. The profiling procedure we outlined seems more fruitful and conceptually satisfying than the traditional matching method. When profiling is not possible, however, it is important to consider the impact of CA confounds and statistical decision procedures used to ensure matching on the control variable, when interpreting syndrome differences on variables of interest.