A further look at the "language-as-fixed-effect fallacy"

Abstract

The proper analysis of experiments using language materials has been a source of controversy and debate among researchers. We summarize the main issues and discuss the solutions that have been presented. Even though the major issues have been dealt with extensively in the literature, there still exists quite a bit of confusion about how to analyze the data from such experiments. We discuss a number of the most frequently voiced objections. In particular, we discuss the issue of what happens if in a counterbalanced design only some of the items show the treatment effect. Finally, a possible solution is discussed for the case where only partial matching of items between conditions is possible.