Generating and confirming hypotheses

Abstract

Dr. Walker asserts that a hypothesis always can be tested using the same data source that generated the data if the test data are independent of the data generating the hypothesis. One way to do this is to use part of the totality of data to generate the hypothesis and the other to test the hypothesis. The validity of this assertion depends on what one means by 'independent'. This note addresses the logical and statistical implications of Dr. Walker's assertion. The key conclusion is that what constitutes 'independent' data has to be considered carefully, and that hypothesis-generating and test data from the same data source generally can not be considered 'independent'.