Scanning the horizon: towards transparent and reproducible neuroimaging research

Abstract

Functional neuroimaging techniques have transformed our ability to probe the neurobiological basis of behaviour and are increasingly being applied by the wider neuroscience community. However, concerns have recently been raised that the conclusions that are drawn from some human neuroimaging studies are either spurious or not generalizable. Problems such as low statistical power, flexibility in data analysis, software errors and a lack of direct replication apply to many fields, but perhaps particularly to functional MRI. Here, we discuss these problems, outline current and suggested best practices, and describe how we think the field should evolve to produce the most meaningful and reliable answers to neuroscientific questions.

Figure 1 |. Sample-size estimates and estimated power for functional MRI studies.

a | A summary of 1,131 sample sizes over more than 20 years, obtained from two sources, is shown: 583 sample sizes were obtained by manual extraction from published meta-analyses by David et al., and 548 sample sizes were obtained by automated extraction from the Neurosynth database with manual verification (for a version with all data points depicted, see Supplementary information S1 (figure)). These data demonstrate that sample sizes have steadily increased over the past two decades, with a median estimated sample size of 28.5 as of 2015. b | Using each of the sample sizes from the left panel, we estimated the standardized effect sizes that would have been required to detect an effect with 80% power for a whole-brain linear mixed-effects analysis using a voxelwise 5% familywise error rate threshold from random field theory (for details, see the main text). The median effect size that the studies in 2015 were powered to find was 0.75. Data and code to generate these figures are available at https://osf.io/spr9a/.

Figure 2 |. Small samples, uncorrected statistics and circularity can produce misleadingly large effects.

A seemingly impressive brain-behaviour association can arise from completely random data through the use of statistics uncorrected for multiple comparisons and circular region-of-interest analyses that capitalize on the large sampling error that arises from small samples. With the informal P < 0.001 and cluster size k >10 thresholding, the analysis revealed a cluster in the superior temporal gyrus (upper panel); the signal extracted from that cluster (that is, using circular analysis) showed a very strong correlation between the functional MRI (fMRI) data and behavioural data (lower panel). For details of the analysis, see the main text. A computational notebook for this example is available at https://osf.io/spr9a/.