Everything Matters: The ReproNim Perspective on Reproducible Neuroimaging

Abstract

There has been a recent major upsurge in the concerns about reproducibility in many areas of science. Within the neuroimaging domain, one approach is to promote reproducibility is to target the re-executability of the publication. The information supporting such re-executability can enable the detailed examination of how an initial finding generalizes across changes in the processing approach, and sampled population, in a controlled scientific fashion. ReproNim: A Center for Reproducible Neuroimaging Computation is a recently funded initiative that seeks to facilitate the "last mile" implementations of core re-executability tools in order to reduce the accessibility barrier and increase adoption of standards and best practices at the neuroimaging research laboratory level. In this report, we summarize the overall approach and tools we have developed in this domain.

Keywords: data model; neuroimaging; publication; re-executability; reproducibility.

FIGURE 1

ReproNim conceptual workflows. (A) Pictorial depiction of the concepts of re-executability (same data, same analysis), replication (same analysis, similar data), robustness (same data, similar analysis), and generalization (similar data, similar analysis). Adapted from multiple sources, including Dr Drummond (2009), Peng (2011); Hong (2015), Goodman et al. (2016); Whitaker (2016), and Allard (2018). (B) General neuroimaging data workflow: Imaging data and behavioral/clinical measures enter into a local analysis, generate results that then get published. Substantial variability in the published literature exists in how the data, analysis and results are described. (C) The ReproNim vision of the general neuroimaging data workflow where control of the data model and machine-readable markup is invoked to completely represent the data workflow, processing and results using the tools of ReproIn, BrainVerse, NICEMAN, and NeuroBlast. (D) Detailed data transformations and markup as the data work their way through the planned analysis and tools.

FIGURE 2

Everything matters. (A) Tools matter: Same data (976 ABIDE1 cases), different tools (FreeSurfer 5.1 and 5.3 and ANTS). For a specific anatomic region (left caudal anterior cingulate cortex), we show a matrix of the between tool comparisons. On the diagonal (from upper left to lower right) we see the distribution histogram of average left caudal anterior cingulate cortex thicknesses for ANTS, FreeSurfer 5.1 and FreeSurfer 5.3, respectively. The three scatter plots (left column, middle, left column bottom and middle column bottom) show the between tool scatter plots and regression line for these data for: ANTS vs. FreeSurfer 5.1 (Pearson’s correlation coefficient r = 0.16); ANTS vs. FreeSurfer 5.3 (Pearson’s correlation coefficient r = 0.21); and FreeSurfer 5.1 vs. FreeSurfer 5.3 (Pearson’s correlation coefficient r = 0.90), respectively. (B) Sample size matters: Same analysis (FreeSurfer 5.3 and a statistical model looking at gender effects in hippocampus volume) as a function of the large-scale publically available structural imaging data in typically developing children in ∼2005 (NIH PEDS, N = 325) and ∼2011 (PING, N = 1239). The plot shows the observed effect size and 95% confidence interval for the total hippocampal volume for these two cohorts. (C) Computational Environment Matters: Same data, same workflow, different workflow operating system environments results in different results, as shown for the volume of the left amygdala in subset of 24 cases. See text for further details.