An Integrated Toolkit for Extensible and Reproducible Neuroscience

Abstract

As neuroimagery datasets continue to grow in size, the complexity of data analyses can require a detailed understanding and implementation of systems computer science for storage, access, processing, and sharing. Currently, several general data standards (e.g., Zarr, HDF5, precomputed) and purpose-built ecosystems (e.g., BossDB, CloudVolume, DVID, and Knossos) exist. Each of these systems has advantages and limitations and is most appropriate for different use cases. Using datasets that don't fit into RAM in this heterogeneous environment is challenging, and significant barriers exist to leverage underlying research investments. In this manuscript, we outline our perspective for how to approach this challenge through the use of community provided, standardized interfaces that unify various computational backends and abstract computer science challenges from the scientist. We introduce desirable design patterns and share our reference implementation called intern.

Fig. 1.

The intern Python library acts as a shock absorber to provide a consistent API to researchers, tool developers, and other users. A community-facing data-access tool should operate on all major data-storage systems (including CloudVolume, DVID, and BossDB), and remain flexible enough to enable common use cases (such as visualization, data upload/download, and data proofreading) without sacrificing performance.

Fig. 2.

Many factors impact data download rate. As an illustration, we tuned the chunk-size parameters for parallel- and non-parallel downloads from the BossDB remote. A. Performance was impacted by client-side compute speed (for data decompression) as well as network throughput, illustrating possible avenues for further abstraction of other remotes. B. Chunked data stores benefit from data requests that are aligned to the cuboid subdivisions in the server backend. This effect is more pronounced in filesystem-based data-stores such as CloudVolume or Zarr, as the cuboid periphery must be downloaded and cropped on local compute resources. In contrast, data stores with cloud-side compute (such as DVID or BossDB) can perform this cropping operation prior to data download, and so the additional egress burden is not incurred.

Fig. 3.

Using the intern library, we downloaded nanometer-resolution 3D imagery and pixel segmentation from the public Witvliet 2020 et al. dataset on BossDB [25]. We then used the meshing service to produce 3D mesh files for visualization in 3D software such as Blender or Neuroglancer. Pictured here is Dataset 2, a C. elegans nematode brain imaged during the L1 larval stage.