Dependency management

[meggart]

To everyone interested in this package. I am currently working on implementing a Zarr backend for LMDB, which would introduce another dependency. There are many more possible backends and compressors and I was wondering how to decide if we should add one to the Zarr base package or to make an extra package that implements the backend in question. Is there already a need for something like a pure Zarr package that only has a few dependencies and where backends and compressors can be loaded on demand? Or shall we just continue adding dependencies as new backends are implemented. Is it an option to just wait until Julia gets a better optional dependency system than what Requires.jl currently offers? Any opinions and suggestions are very welcome.

[meggart]

Starting with Julia 1.9 we will have the possibility to define optional dependencies and can potentially move some of our hard dependencies to the weakeps section. However, there still remains the question which compressors and backends should be available by default through using Zarr and which ones should be available only the respective module is loaded.

Currently we load AWSS3, HTTP, Blosc and Zlib by default and I have an unregistered LMDBStore package I would like to integrate somehow. In addition, there is a suggestion to make an fsspec-based store that is using PythonCall and some python deps in the background.

All suggestions how to organize this are welcome.

[agoodm]

In general I think it makes a lot of sense to make additional storage types beyond the most basic ones optional dependencies. I know zarr-python manages this through conditional imports.

I would say of the four you mentioned maybe AWSS3 could be moved to being optional, and likewise for any proposed fsspec compatible store.