Are custom stores supported?

[Ozaq]

I have been working on creating a custom store despite there being no documentation. Upgrading from 3.0.x to 3.1.x now broke my store, not nice, but let's be fair I am not really working against an officially supported interface (because not documented).

Which brings me to the point: Can you please clarify the documentation w.r.t. custom stores? Does "Coming soon" imply that the documentation is missing or that there is currently no support / desire to allow custom stores?

Thanks.

[jhamman]

Hi @Ozaq -- sorry the documentation was not clear here and that you hit some issues with 3.1. A few comments and questions:

1. What broke with the 3.1 upgrade? As far as I know, there were no breaking changes to the store interface in that release.
2. "Are custom stores supported?" -- Yes, absolutely! At Earthmover, we've developed Icechunk as a custom zarr store. Anything that fullfills the StoreABC and passes the store tests should work
3. Finally, I'm curious to learn a bit about what the custom store your working on is all about!

[d-v-b]

The store ABC is documented, but maybe we should make this more prominent? We could also add a self-contained example that implements a custom store from scratch so people can see how it works.

[Ozaq]

Hi @jhamman for getting back to me

1. What broke with the 3.1 upgrade? As far as I know, there were no breaking changes to the store interface in that release.

We had been omitting fill_value=... from our .zarr json because the arrays we generate do not use fill values. Until 3.0.10 this worked and starting with 3.1.0 this resulted in no data being read from our store. I am aware that this was our mistake in not following the zarr v3 spec.

2. "Are custom stores supported?" -- Yes, absolutely! At Earthmover, we've developed Icechunk as a custom zarr store. Anything that fullfills the StoreABC and passes the store tests should work

I was not aware of those test cases, this should help us a lot to ensure we are implementing the interface correctly. I guess we are still missing parts.

3. Finally, I'm curious to learn a bit about what the custom store you're working on is all about!

I am working on providing Zarr from FDB, a domain specific object store for Earth Systems Data [1]. Our Zarr store implementation can be found here: Fdb/zarr-fdb-interface / Store implementation

The intended use case is: Your data exceeds your storage capability and you already have GRIB data, now you can access the data encoded in GRIB "as-if" it would be zarr.

The project is in a pre-MVP state.

@d-v-b A concise end-to-end example including how to apply the store-tests would have been great. I agree that a lot of the information is there but it is a problem of discoverability. Also labeling custom stores as "Coming Soon" is confusing.

I understand that much of my issues stem from me not being familiar with the python-zarr code and my ramp-up time could have been much smoother with more in depth "here is how to do it" style documentation.

Edit: Grammar

[d-v-b]

I see a few areas for improvement here on our side:

• A simple explanation of what's inside the zarr metadata documents (zarr.json, .zattrs, etc). This would be docs and also python types (I think a typeddict for each metadata documents would be useful)
• A better definition of what a "store" is, and better docs for explaining how to implement + test a custom store
• Overall, better API docs that make this stuff easier to discover

[sehoffmann]

Minor comment: zarr.abc.store.Store isn't exposed as module / top-level api at the moment. I.e. I must do from zarr.abc.store import Store for any code that interacts directly with stores (e.g. as type hint). Would be nice if there was just a zarr.Store symbol that I could use right out of the box. I suppose this applies to all ABCs.

For instance, this code looks quite cumbersome:

    @property
    def store(self) -> zarr.abc.store.Store:
        """
        Underlying Zarr store
        """
        return self._root.store