Clarifying questions regarding building xtensor-zarr

[abhibaruah]

Hello,

I had a few questions regarding using xtensor-Zarr for supporting reading/writing Zarr files and I was hoping someone in the xtensor-zarr dev team can help us with that.

1. "xtensor-zarr depends on xtensor ^0.23.8, xtensor-io ^0.12.7, zarray ^0.0.7, nlohmann_json ^3.9.1, Blosc ^1.21.0, zlib ^1.2.11, gdal ^3.0.0 and cpp-filesystem ^1.3.0" a. Are any of these dependencies optional? b. Is gdal needed? If yes, does gdal need to be built with Virual File System? Can you elaborate a little more on the need for gdal in the upstream dependencies?
2. "xtensor-zarr comes with a libxtensor-zarr-gdal shared library for accessing stores through GDAL's Virtual File System or the local file system." Does this mean that we can simply use the libxtensor-zarr-gdal shared library and dont need gdal shared libraries separately?
3. Is there support for reading and writing to files stored in Microsoft Azure Blob Storage?
4. Are there any limitations / caveats with xtensor-zarr compared to the Zarr Python library?

I look forward to hearing back from you.

[joshmoore]

I'll be interested to hear the answer to your questions, @abhibaruah, but two additions from my side:

• 1. I recently needed to pin gdal to get xtensor-zarr to install -- https://github.com/zarr-developers/zarr_implementations/pull/57/commits/61491e27d0c18e665e6680d3bfcce8b949ee9c9f
• 4. the zarr v3 spec has been updated recently which is unlikely to be reflected here yet (see https://zarr.dev/blog/zep1-update/)

[abhibaruah]

Thanks @joshmoore. Would love to hear more from the xtensor-Zarr team regarding the dependency on gdal. Did you have to build gdal with Virtual File System to build? I would also be curious to know if the recent updates to the zarr v3 spec would be implemented in a future release of xtensor-zarr.

[abhibaruah]

@dzenanz I was wondering if you can help in answering my questions above. I havent been able to get any response from anyone in the Quantstack team. I apologize for tagging you here in case you are not the right person to answer the questions above. I see that you have some experience working with xtensor-zarr. Would you be able to provide any insight into the questions above?

[dzenanz]

I am working on ITK IO for zarr (specifically, OME-NGFF). Initially, I used xtensor-zarr (PR6), but then realized that xtensor-zarr only supports some version of zarr v3 draft. I then pivoted to netCDF, to be bogged down in its build system. Finally I settled on tensorstore. The only remaining issue is building Python packages.

xtensor-zarr seems to be immature project, and no longer supported.

[abhibaruah]

Thanks a lot @dzenanz. This is really helpful. We were exploring xtensor-zarr and netCDF simultaneously. If you dont mind me asking, can you tell me a bit more regarding the netCDF build problems you were facing. Were they related to NCZarr and building with aws-sdk-cpp, or general netCDF building (I understand that building netCDF from source is a little overwhelming) ?

[dzenanz]

I guess it is more general netCDF building. More specifically difficulty of integrating it into another library via CMake.

[dzenanz]

Here are some issues I opened there: https://github.com/Unidata/netcdf-c/issues?q=is%3Aissue+sort%3Aupdated-desc+author%3Adzenanz+

This one is zarr-specific.

[abhibaruah]

Thanks once again. This was indeed helpful.