NeuroWRAP: integrating, validating, and sharing neurodata analysis workflows

Abstract

Multiphoton calcium imaging is one of the most powerful tools in modern neuroscience. However, multiphoton data require significant pre-processing of images and post-processing of extracted signals. As a result, many algorithms and pipelines have been developed for the analysis of multiphoton data, particularly two-photon imaging data. Most current studies use one of several algorithms and pipelines that are published and publicly available, and add customized upstream and downstream analysis elements to fit the needs of individual researchers. The vast differences in algorithm choices, parameter settings, pipeline composition, and data sources combine to make collaboration difficult, and raise questions about the reproducibility and robustness of experimental results. We present our solution, called NeuroWRAP (www.neurowrap.org), which is a tool that wraps multiple published algorithms together, and enables integration of custom algorithms. It enables development of collaborative, shareable custom workflows and reproducible data analysis for multiphoton calcium imaging data enabling easy collaboration between researchers. NeuroWRAP implements an approach to evaluate the sensitivity and robustness of the configured pipelines. When this sensitivity analysis is applied to a crucial step of image analysis, cell segmentation, we find a substantial difference between two popular workflows, CaImAn and Suite2p. NeuroWRAP harnesses this difference by introducing consensus analysis, utilizing two workflows in conjunction to significantly increase the trustworthiness and robustness of cell segmentation results.

Keywords: consensus; image analysis; reproducibility; two-photon calcium imaging; workflow management.

FIGURE 1

Example workflows within NeuroWRAP. These three example workflows contain modules available within NeuroWRAP. Color denotes type of processing step categorized by the column headers. Workflow 3 uses results from a previous execution to calculate dF/F using an alternative method followed by downstream analysis.

FIGURE 2

Example workflow execution. An analysis workflow featuring 4 modules. Below each module are its input and output variables, where required inputs are bolded and optional inputs are italicized. Dashed arrows represent output variables that have been piped as inputs to downstream modules. All pictured information is stored in the output file upon execution of this workflow. On the right, a screenshot of the NeuroWRAP GUI is shown illustrating the setup page for this specific workflow.

FIGURE 3

Results are sensitive to algorithm choice and parameter configuration. (A) Example of the cell detection consensus module in NeuroWRAP. The top schematic illustrates an example usage of the cell segmentation consensus module. The lower image is example output of the consensus module using Suite2p and CaIman on publicly available Neurofinder data. (B) Number of cells detected by Suite2p and CaImAn while altering certain input parameters. Top left: “threshold_scaling,” top right: “max_overlap” (Suite2p), bottom left: “gSig” (CaImAn), bottom right: number of expected cells in field of view extrapolated from “K” (CaImAn). (C) Left: Number of cells detected by CaImAn across a selected parameter space. Middle: Ratio of detected cells in CaImAn to Suite2p across a selected parameter space. White pixels indicate parameter configurations where roughly the same number of neurons were detected. Right: Percentage of cells that were spatially matched between Suite2p and CaImAn at various parameter configurations. (D) Cell-location consensus between ground truth data and CaImAn’s detected cells across a selected parameter space.

FIGURE 4

Cell detection comparisons to ground truth data. (A) Proportion of CaImAn’s detected cells that overlap with ground truth cell locations at a range of parameter configurations. (B) Proportion of consensus analysis (Suite2p and CaImAn) cell locations that overlap with ground truth cell locations at a range of parameter configurations. (C) Scatter plot of each pixel value in panels plots (A,B), comparing the percentage overlap with ground truth from CaImAn alone to that of CaImAn’s consensus with Suite2p.