OME-Zarr: a cloud-optimized bioimaging file format with international community support

Abstract

A growing community is constructing a next-generation file format (NGFF) for bioimaging to overcome problems of scalability and heterogeneity. Organized by the Open Microscopy Environment (OME), individuals and institutes across diverse modalities facing these problems have designed a format specification process (OME-NGFF) to address these needs. This paper brings together a wide range of those community members to describe the cloud-optimized format itself-OME-Zarr-along with tools and data resources available today to increase FAIR access and remove barriers in the scientific process. The current momentum offers an opportunity to unify a key component of the bioimaging domain-the file format that underlies so many personal, institutional, and global data management and analysis tasks.

Keywords: Bioimaging; Cloud; Community; Data; FAIR; Format.

Fig. 1

A common format enables a diverse set of use cases via a consistent API. A wide range of modalities can be converted into a representation that can be equally accessed by a variety of tools. This format can be used to download entire datasets for local processing, to stream pyramidal sub-resolutions for interactive viewing or to process entire resolutions in parallel. OME-Zarr data shown includes a idr0076 (Ali et al. 2020), b idr0101 (Payne et al. 2021), c idr0077 (Valuchova et al. 2020), d S-BIAD548 (Lim et al. 2023), e S-BIAD217 (de Boer et al. 2020), and f S-BIAD501 (Igarashi et al. 2015)

Fig. 2

By making use of an annotated hierarchy of arrays, OME-Zarr can represent complex relationships between images, capture the multiple resolutions of an image pyramid, and provide tunable chunk size and compression all within a single abstraction layer that can be saved as a directory of files on disk or shared remotely. a Each level of nested directories provides a different level of abstraction: the top-level directory can represent an entire 1 Terabyte plate with more than 100,000 pixels in the X and Y dimensions, while the lowest level directory represents individual chunks of N-dimensional data as small as 1 Megabyte. b In the example shown, a concatenation of low-resolution images produces a 2560 pixels × 1822 pixel representation of the entire plate, followed by similar examples of how many pixels must be loaded by a client at each zoom level

Fig. 3

The original viewers of OME-Zarr published in Moore et al. , from left to right BigDataViewer, napari, and Vizarr, here seen loading a view of the same EM volume of a 6 day old Platynereis larva from (Vergara et al. 2020) available at https://s3.embl.de/i2k-2020/platy-raw.ome.zarr. These three applications provided broad coverage over the most common bioimaging platforms like Fiji and napari but critically also a web viewer that could stream data on the fly

Fig. 4

Advanced GPU Accelerated Volume Explorer (AGAVE) displaying a downsampled level from a multi-terabyte mouse brain OME-Zarr dataset. The number of pixels actually loaded is displayed at lower right. The full resolution data is 47,310 × 20,344 × 18,471 which consumes about 33 TB. The ability to quickly access multiresolution data makes low latency interactive visualization possible

Fig. 5

ITKWidgets 3D rendering an OME-Zarr for IDR 0062A in Jupyter. Interactive features shown include volume rendering, slicing planes, and interactive widgets to adjust rendering parameters and slice planes indices

Fig. 6

Neuroglancer rendering the same OME-Zarr from IDR 0062A as Fig. 5

Fig. 7

OME-NGFF Validator validating the same image from IDR 0062A on the left, and on the right providing a summary of the size of the data as well as providing a quick visualization of a single plane

Fig. 8

webknossos loading an EM volume of a 6 day old Platynereis larva from (Vergara et al. 2020) with a manually added segmentation. The web accessible version is accessible at https://wklink.org/6422

Fig. 9

Allen Cell Volume Viewer displaying a multichannel fluorescence image of gene edited hiPSC cells via a downsampled level of an OME-Zarr converted from the dataset found at https://cfe.allencell.org/?dataset=variance_v1

Fig. 10

Example application of BigStitcher’s interest point based registration on one of the first “exaSPIM” lightsheet microscope datasets acquired at the Allen Institute for Neural Dynamics (sample 609,281, available at the link in Table 2). The overlapping regions of two tiles are shown in BDV at their nominal (left) and aligned (right) locations. This large scale NGFF dataset consists of 54 tiles with dimensions of 24,576 × 10,656 × 2048 voxels (about 1 TB raw size) each

Fig. 11

Visualization of a MERFISH mouse brain dataset [Allen Institute prototype MERFISH pipeline (Long et al. 2023)] via the napari-spatialdata plugin, featuring single-molecule transcripts (points) and their rasterized representation (image), polygonal ROIs, and annotated cells approximated as circles with variable radii. The dataset has been converted to OME-Zarr with the SpatialData APIs

Fig. 12

NGFF-Converter GUI showing a sample of input formats being converted to OME-Zarr

Fig. 13

An OME-Zarr dataset on DANDI (ID: DANDI:000108) contains multiscale 5D datasets (time, channel, z, y, x), with metadata, scale, and transformations. a DANDI:000108 includes multi-slab, multi-stain (NeuropeptideY-NPY, Calretinin-CR, YOYO1) data that are aligned and the coordinate transforms are stored in the Zarr files, allowing b on-the-fly visualization and stitching of the slabs at multiple scales (two shown) using Neuroglancer. c An HTML dashboard allows data submitter and any user on the Web to see the live status: samples + stains uploaded, their quality issues, and multiscale viewing of this 300 + TB dataset

Fig. 14

a ‘Imaging’ tab of the zebrahub.org website showing the list of imaging datasets made available. b By clicking on a particular dataset, the user is directed to a neuroglancer instance that allows interactive exploration of the dataset

Fig. 15

Multiscale OME-Zarr representation of a whole brain dataset (https://download.brainimagelibrary.org/2b/da/2bdaf9e66a246844/mouseID_405429-182725/) (~ 6 TB compressed) archived at BIL and visualized over the internet in napari via the napari-ome-zarr plugin. Data is stored at BIL in an alternative format and is dynamically converted to OME-Zarr chunk-by-chunk and delivered to clients upon request