Zarr

Zarr is a cloud-native, chunked, compressed, and hierarchical array data format.

Contents

Resources

• Existing resources
• Introductory videos
• Zarr V3
• Libraries
• Platforms
• Articles
• Talks & Videos
• Life sciences

Topics

• Zarr & other array data formats
• GeoZarr
• Zarr & STAC

Resources

Existing resources

The Zarr website is already an excellent resource for learning about Zarr and its ecosystem. This list is intended to complement the website with a curated and opinionated list of resources.

This list focuses on Geo/Earth Sciences, but is not limited to that domain.

Existing lists

Lists

• The Zarr website already contains great lists: Zarr Implementations, Zarr Datasets, Zarr metadata conventions
• Zarr tutorials (zarr-developers/tutorials)
• Projects using Zarr (zarr-developers/community#19)
• Beautiful Zarr (zarr-developers/beautiful-zarr)
• See playlists & lists in Talks & Videos

Introductory videos

Introductory talks Youtube playlist

Two excellent and up-to-date introductory talks:

• Sanket Verma: The Beauty of Zarr
• Ryan Abernathey: State of Zarr

Zarr V3

Zarr V3 is the upcoming version of Zarr. It is a major update that will bring many new features and improvements.

If you're getting into Zarr now, it might be a good idea to start with Zarr V3.

• Zarr-Python 3 and why you should be excited!

For an excellent in-depth overview, see the ESIP series of talks

• 2023-03-27 ESIP Cloud Computing Cluster: Zarr - The Next Generation
• 2023-04-24 ESIP Cloud Computing Cluster: Next Generation of Zarr Part 2/3 GeoZarr and Zarr Sharding
• 2023-05-22 ESIP CCC: Next Gen Zarr Part 3/3: accumulation proposal, Kerchunk and Pangeo-Forge

Libraries

This list contains libraries that directly relate to Zarr in some way.

For implementations of Zarr, see Zarr Implementations.

• kerchunk, see kerchunk section
• xpublish: Exposing as and consuming Zarr through a REST API
  • See also routers at xpublish-community, e.g. xpublish-opendap
  • Improving Access to NOAA NOS Model Data with Kerchunk and Xpublish
• ndpyramid: utility for generating ND array pyramids using Xarray and Zarr

Storage & I/O

• Tensorstore and xarray-tensorstore: library for efficiently reading and writing large multi-dimensional arrays, has Zarr API
• KivkIO: C++ and Python bindings to cuFile, enabling GPUDirect Storage
• rechunker: disk-to-disk transformation for chunked arrays
• xpartition: writing large xarray datasets to Zarr. Works around shortcomings of Dask (distributed#6360)

ETL

• Xarray: Zarr is commonly written and accessed through xarray's API.
  • Xarray has its own Zarr Encoding Specification
• xarray-beam: Integration of xarray and Apache Beam built using Zarr.
• Pangeo-forge: Open-source data platform for transforming datasets into analysis-ready cloud-optimized formats.
  • See Pangeo Forge in 4 minutes and Pangeo Forge: Crowdsourcing Open Data in the Cloud- Ryan Abernathey | SciPy 2022

Developer-oriented

• numcodecs: Compression and transformation codecs used by Zarr
• pydantic-zarr: Pydantic models for Zarr objects
• traverzarr: Traversing Zarr JSON as if it's a filesystem
• zarr_checksum: Calculating checksum information form Zarr
• zarrdump: Describe zarr stores from the command line

Visualization: For tools & libraries for visualization, see visualization section

Kerchunk

Kerchunk allows you to efficiently read chunked data formats such as GRID, NetCDF, COGs by exposing them as a Zarr store.

Talks and tutorials

• All you need is Zarr
• 2022 ESIP Kerchunk Tutorial
• Accessing NetCDF and GRIB file collections as cloud-native virtual datasets using Kerchunk

Future of Kerchunk

In the future, Kerchunk will be split into upstream functionality in Zarr itself and a new VirtualiZarr package.

• Kerchunk JSON references will become a part of the Chunk manifest
• For a full overview, see Upstreaming Kerchunk
• What's Next for Kerchunk

Platforms

• Arraylake: a data lake platform based on Zarr. The company, Earthmover was started by core Zarr developers.

Articles

• NASA IMPACT: Zarr Visualization Report
• Earthmover: cloud-native data loaders for machine learning using zarr and xarray
• Zarr Sprint Recap relevant overviews

Talks & Videos

Existing lists

• Zarr Developers playlists, namely
  • Zarr: Introductory Talks
  • Zarr: Zarr: Projects, Uses, Research and Workflows
• Zarr Talks
• Introductory videos in this list

Talks

• Earthmover Webinar: Building a Planetary Scale Earth Observation Data Cube in Zarr with code repository and slides
• Earthmover Webinar: Analysis-ready Weather Forecast Data Cubes with Zarr with code repository and slides
• Presentation | Zarr: Community specification of large, cloud-optimised, N-dimensional, typed array storage
• Presentations for Sanket Verma's talks: SciPy 2023 and PyCon DE 2023

Life sciences

Zarr has seen great adoption in the life sciences domain.

• bdz: Zarr-based format for storing quantitative biosystems dynamics data
• ome-zarr-py: Implementation of next-generation file format (NGFF) specifications for storing bioimaging data in the cloud.
• ez_zarr: Easy, high-level access to OME-Zarr filesets
• hdmf-zarr: Zarr I/O backend for HDMF

Talks and resources

• Zarr | Life Science Lightning Talk | Trevor Manz | Dask Summit 2021
• Accelerating Single-cell Bioinformatics with N-dimensional Arrays in the Cloud | ISMMS
• What are next-generation file formats (NGFF)?

Visualization

Zarr has seen most work on visualization in the bioimaging community:

• List: Image viewers with OME-Zarr support
• WEBKNOSSOS: web-based visualization & annotation tool, supports OME-Zarr
• Napari: interactive viewer
• Vizarr: interactive viewer built using viv (OME-Zarr and OME-TIFF)
• Neuroglancer: WebGL-based viewer for volumetric data
• BigDataViewer

Topics

Zarr & other array data formats

For a general overview, see

• Introduction to Cloud-Native Geospatial Formats
• Cloud-Optimized Geospatial Formats Guide.

Essentially all other common array data formats can be exposed as Zarr. See Kerchunk.

NetCDF & HDF5

Zarr, NetCDF, and HDF5 are three separate data formats that nonetheless relate to each other in multiple ways.

• Zarr inherits its hierarchical structure from HDF5.
• Zarr is commonly accessed through xarray, whose data models are based on the NetCDF data format
• NetCDF4 can use HDF5 as a backend
• NCZarr is an extension of the Zarr format to map it to a subset of the NetCDF data model.

Resources

• A Comparison of HDF5, Zarr, and netCDF4 in Performing Common I/O Operations HDF5
• Pangeo: HDF5 at the speed of Zarr
• Joe Jevnik: Zarr vs. HDF5 | PyData New York 2019

COG: Cloud-Optimized GeoTIFF

• Overview: Zarr - A Cloud Native ND Array Format @ 19:17
• COG and Zarr for Geospatial Data

N5

Zarr and N5 are two similar array data formats that share common goals and development.

The Zarr V3 spec aims to provide a common implementation target (sources: 1, 2)

Links

• n5
• zarr.n5
• z5: C++ and Python interface for datasets in zarr and n5 format
• Zarr N5 spec diff (zarr-specs#3)

GeoZarr

GeoZarr is a proposal for a Zarr-based geospatial data format, being submitted as an OGC standard

GeoZarr will define a metadata convention for Zarr stores that contain geospatial data.

It will also define the relationship of Zarr with CF and NetCDF

Links

• Specs
• Current status of GeoZarr

Zarr & STAC

STAC provides a common structure for describing and cataloging spatiotemporal assets.

With its hierarchical structure and key-value metadata support, Zarr's capabilities overlap significantly with STAC.

The communities have not yet converged on a canonical representation of Zarr datasets through STAC.

Today, a good example of exposing Zarr in STAC is Planetary Computer

• Reading Zarr Data
• STAC collection: Daymet Annual North America
• STAC collection: CIL Global Downscaled Projections for Climate Impacts Research
• xstac: STAC from xarray
• Related STAC extensions: xarray-assets, datacube

More discussion & Related links

• Pangeo: Metadata duplication on STAC zarr collections
• geozarr-spec#32: Integration of Zarr with STAC Catalogs
• stac-spec#781: Zarr Extension?
• Tom Augspurper: STAC and Kerchunk
• Presentation | Daniel Jahn – STAC vs Zarr
• Arraylake a data lake platform that is arguably the first example of a pure Zarr data catalog

In the future, the Zarr V3 Spec and GeoZarr convention will likely enable greater interoperability between STAC and Zarr.