Cross-validation and hypothesis testing in neuroimaging: An irenic comment on the exchange between Friston and Lindquist et al

Abstract

The "ten ironic rules for statistical reviewers" presented by Friston (2012) prompted a rebuttal by Lindquist et al. (2013), which was followed by a rejoinder by Friston (2013). A key issue left unresolved in this discussion is the use of cross-validation to test the significance of predictive analyses. This note discusses the role that cross-validation-based and related hypothesis tests have come to play in modern data analyses, in neuroimaging and other fields. It is shown that such tests need not be suboptimal and can fill otherwise-unmet inferential needs.

Keywords: Brain decoding; Cross-validation; Likelihood ratio test; Neyman–Pearson Lemma; Null hypothesis; Permutation test.

Figure 1

(a) Estimate of β obtained by minimizing the Lasso criterion (7) with log(λ) = −6. The rows represent 21 pain-relevant regions, while the columns represents 23 time points. The first dashed line marks the end of the hot stimulus, while the second marks the time at which “How painful?” appeared on the screen. (b) Estimate based on log(λ) = −1.84. (c) Ten-fold cross-validated mean squared error, ± 1 standard error (SE), for a range of log(λ) values. The first dotted line indicates where CV is minimized (log(λ) = −1.84); the second indicates the largest log(λ) at which the CV score is within 1 SE of the minimum. The numbers along the top edge are counts of nonzero coefficient estimates, which decrease as λ increases. (d) Histogram of 999 permuted-data CV scores, with the real-data CV score indicated by the red line.

Figure 2

Power of the CV permutation test (black) compared with that of the benchmark F-test (green), for two cases with p = 2: a one-way ANOVA comparing two groups (above) and a normally distributed covariate (below).