Tom Nicholas

Listing every format that could be represented as virtual zarr

19

**Let's list all the file formats that could potentially be represented efficiently as "virtual zarr" - i.e. zarr + chunk manifests.** The important criteria here is that the format must...

Executable code examples in docs

2

The main usage docs revolve around code examples, but these are currently just hard-coded in. It would be good to generate these results automatically, and even better if the docs...

Virtualize ZIP archives?

1

[This article](https://medium.com/@mojodna/tapalcatl-cloud-optimized-tile-archives-1db8d4577d92#:~:text=ZIPs%20facilitate%20random,file%20of%20interest.) is fascinating: > ZIPs facilitate random access and cheap updates as part of their design. > > Random access is achieved by providing a [central directory](https://en.wikipedia.org/wiki/Zip_%28file_format%29#Structure), which contains...

Document use as a runtime translation layer

6

I realized there is now another use case for VirtualiZarr that is not clearly documented: using `ManifestStore` as a runtime translation layer to access other cloud-optimized formats (e.g. COGeoTIFF) through...

Test endianess support against zarr-python data types refactor

4

This PR is for testing support for @d-v-b's unmerged zarr data types refactor in https://github.com/zarr-developers/zarr-python/pull/2874. Particularly we care about supporting big-endian data. - [ ] Closes #xxxx - [ ]...

- [x] Closes #132 - [ ] Changes are documented in `docs/releases.rst` A list of everything about this demo that is janky and should be fixed before merging, in order...

This (WIP) PR adds a `to_icechunk()` method directly on `ManifestGroup`, and refactors the `.to_icechunk()` virtual dataset accessor method to go via the new `ManifestGroup` method. The point is 1. To...

Documents features added in #349, and adds an entirely new docs page on Scaling. - [x] Changes are documented in `docs/releases.rst` - [x] New functions/methods are listed in `api.rst` -...

Aims to refactor all the kerchunk-based readers to use `ManifestStore`, but to do that this PR first adds a `ManifestStore.from_kerchunk_refs()` method. - [ ] Closes #xxxx - [ ] Tests...

Use obstore / obspec for globbing in `open_virtual_mfdataset`

4

### Problem `xarray.open_mfdataset` accepts a string with wildcards, and then uses this [janky bit of fsspec code](https://github.com/pydata/xarray/blob/888dfdbeabc5ed1be201f9e9a895a6d0a21ae6d7/xarray/backends/common.py#L146) to glob with it. But it's pretty fragile - in particular it confusingly...