Nonparametric permutation tests for functional neuroimaging: a primer with examples

Abstract

Requiring only minimal assumptions for validity, nonparametric permutation testing provides a flexible and intuitive methodology for the statistical analysis of data from functional neuroimaging experiments, at some computational expense. Introduced into the functional neuroimaging literature by Holmes et al. ([1996]: J Cereb Blood Flow Metab 16:7-22), the permutation approach readily accounts for the multiple comparisons problem implicit in the standard voxel-by-voxel hypothesis testing framework. When the appropriate assumptions hold, the nonparametric permutation approach gives results similar to those obtained from a comparable Statistical Parametric Mapping approach using a general linear model with multiple comparisons corrections derived from random field theory. For analyses with low degrees of freedom, such as single subject PET/SPECT experiments or multi-subject PET/SPECT or fMRI designs assessed for population effects, the nonparametric approach employing a locally pooled (smoothed) variance estimate can outperform the comparable Statistical Parametric Mapping approach. Thus, these nonparametric techniques can be used to verify the validity of less computationally expensive parametric approaches. Although the theory and relative advantages of permutation approaches have been discussed by various authors, there has been no accessible explication of the method, and no freely distributed software implementing it. Consequently, there have been few practical applications of the technique. This article, and the accompanying MATLAB software, attempts to address these issues. The standard nonparametric randomization and permutation testing ideas are developed at an accessible level, using practical examples from functional neuroimaging, and the extensions for multiple comparisons described. Three worked examples from PET and fMRI are presented, with discussion, and comparisons with standard parametric approaches made where appropriate. Practical considerations are given throughout, and relevant statistical concepts are expounded in appendices.

Figure 1

Histogram of permutation distribution for single voxel using a mean difference statistic. Note the symmetry of the histogram about the y‐axis. This occurs because for each possible labeling, the opposite labeling is also possible, and yields the same mean difference but in the opposite direction. This trick can be used in many cases to halve the computational burden.

Figure 2

Mesh plots of parametric analysis, z = 0 mm. Upper left: slope estimate. Lower left: standard deviation of slope estimate. Right: t image for DURATION. Note how the standard deviation image is much less smooth than slope image, and how t image is correspondingly less smooth than slope image.

Figure 3

Mesh plots of permutation analysis, z= 0 mm. Upper left: Slope estimate. Lower left: square root of smoothed variance of slope estimate. Right: pseudo t image fot=r DURATION. Note that smoothness of pseudo t image is similar to that of the slope image (c.f. figure 2).

Figure 4

A: Distribution of maximum suprathreshold cluster size, threshold of 3. Dotted line shows 95^th percentile. The count axis is truncated at 100 to show low‐count tail; first two bars have counts 579 and 221. B: Maximum intensity projection image of significantly large clusters.

Figure 5

A: Permutation distribution of maximum repeated measures t‐statistic. Dotted line indicates the 5% level corrected threshold. B: Maximum intensity projection of t‐statistic image, thresholded at critical threshold for 5% level permutation test analysis of 8.401.

Figure 6

A: Test significance (α) levels plotted against critical thresholds, for nonparametric and parametric analyses. B: Maximum intensity projection of t image, thresholded at parametric 5% level critical threshold of 11.07.

Figure 7

A: Permutation distribution of maximum repeated measures t statistic. Dotted line indicates the 5% level corrected threshold. B: Maximum intensity projection of pseudo t statistic image threshold at 5% level, as determined by permutation distribution. C: Maximum intensity projection of t statistic image threshold at 5% level as determined by permutation distribution. D: Maximum intensity projection of t statistic image threshold at 5% level as determined by random field theory.