Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2022 Aug 1;35(4):475-481.
doi: 10.1097/WCO.0000000000001081.

Reproducibility and replicability in neuroimaging data analysis

Tü Lay Adali  1 , Vince D Calhoun  2
Affiliations
Review

Reproducibility and replicability in neuroimaging data analysis

Tü Lay Adali et al. Curr Opin Neurol. .

Abstract

Purpose of review: Machine learning solutions are being increasingly used in the analysis of neuroimaging (NI) data, and as a result, there is an increase in the emphasis of the reproducibility and replicability of these data-driven solutions. Although this is a very positive trend, related terminology is often not properly defined, and more importantly, (computational) reproducibility that refers to obtaining consistent results using the same data and the same code is often disregarded.

Recent findings: We review the findings of a recent paper on the topic along with other relevant literature, and present two examples that demonstrate the importance of accounting for reproducibility in widely used software for NI data.

Summary: We note that reproducibility should be a first step in all NI data analyses including those focusing on replicability, and introduce available solutions for assessing reproducibility. We add the cautionary remark that when not taken into account, lack of reproducibility can significantly bias all subsequent analysis stages.

PubMed Disclaimer

Conflict of interest statement

Conflicts of interest

None.

Figures

Figure 1:
Figure 1:
Functional network connectivity (FNC) maps for (a) run with the lowest Cross-ISI (best run); and (b) run with the highest Cross-ISI (low reproducibility). Note that the best run result has better interpretability. (AUD: Auditory; MOT: Sensorimotor; VIS: Visual; ATTN: Attentional, and FRONT: Frontal networks; DMN Default mode network.)
Figure 1:
Figure 1:
Functional network connectivity (FNC) maps for (a) run with the lowest Cross-ISI (best run); and (b) run with the highest Cross-ISI (low reproducibility). Note that the best run result has better interpretability. (AUD: Auditory; MOT: Sensorimotor; VIS: Visual; ATTN: Attentional, and FRONT: Frontal networks; DMN Default mode network.)
Figure 2:
Figure 2:
Variability in volume (max-min) and percent signal change in various brain regions produced by FreeSurfer while using different random seeds.
See this image and copyright information in PMC

References

    1. National Academies of Sciences, Engineering, and Medicine, Reproducibility and Replicability in Science. Washington, DC: The National Academies Press, 2019. [Online]. Available: https://www.nap.edu/catalog/25303/reproducibility-and-replicability-in-science

      ** The reference solidifies the definitions for reproducibility and replicability along with other related concepts and provides a solid reference in the area.

    1. Adali T, Kantar F, Akhonda MABS, Strother SC, Calhoun VD, and Acar E, “Reproducibility in matrix and tensor decompositions: Focus on model match, interpretability, and uniqueness,” IEEE Signal Processing Magazine, 2022.

      ** The paper addresses reproducibility in matrix and tensor decompositions (MTD) that have been growing in importance in the analysis of neuroimaging data. Authors make use of two widely used methods with relaxed uniqueness guarantees, independent component analysis, and the canonical-polyadic decomposition, and provide examples to solidify these concepts and demonstrate the tradeoffs in practical applications of MTD. Finally, a reproducibility checklist for MTDs is provided similar to those developed for supervised learning.

    1. Group ICA of fMRI toolbox: http://trendscenter.org/software/gift/. [Online]. Available: http://trendscenter.org/software/gift/

      * The toolbox incorporates multiple methods to assess the reliability of the solutions.

    1. (2022) FreeSurfer.[Online]. Available:https://surfer.nmr.mgh.harvard.edu
    1. (2022) The Turing Way Handbook.[Online].Available:https://the-turing-way.netlify.app/

Publication types