Ryan Abernathey
Ryan Abernathey
This is a good idea David. Perhaps we could store the list of input files directly in the target dataset metadata itself (`attrs`). This would be useful for incremental appending...
The input hashing stuff introduced by @cisaacstern in #349 should make this doable. The user story for this is being tracked in https://github.com/pangeo-forge/user-stories/issues/5. Charles, would you be game for diving...
That sounds reasonable to me. I think in general we should be injecting extra metatdata into the datasets we write. Stuff like ``` { "pangeo-forge:version": 0.6.2, "pangeo-forge:recipe-hash": "a1b2c3", "pangeo-forge:input-hash": "..."...
My understanding has been that if I call `zarr.open(fs.get_mapper())`, the FSMap would be coerced to an FSStore. If this is not the case, we should fix it! Having these two...
> We should therefore probably add both from_filesystem and from_mapper class methods to support all possible inputs but always end up with the same instance type. I would def be...
> I think having both `fsspec.mapping.FSMap` and `zarr.storage.FSStore` is confusing. They both implement the mutable mapping interface and so both can be passed to `zarr.open`, but one might be better...
> One intermediate option might be to add a class method to [zarr.storage.FSStore](https://github.com/zarr-developers/zarr-python/blob/af0c36b2a612579db6d4ecc05388d9dcb2b3b1c2/zarr/storage.py#L1085) to allow you to initialize an FSStore from an existing fsspec.AbstractFileSystem + a path. I am starting...
With https://github.com/zarr-developers/zarr-python/pull/911 we should be able to add the following method to `FSSpecTarget` ```python def get_fsstore(self) -> zarr.storage.FSStore: """Get a Zarr FSStore object suitable for storing Zarr data.""" return zarr.storage.FSStore(self.root_path,...
I am trying the binders now. Everything seems to work. Two suggestions: - Have the binders open in jupyterLab, not classic notebook. - Add a general binder link to the...
Comprehensive test suite FTW! The test failure is due to HTTP credentials not being passed through via fsspec_open_kwargs. Here is a minimal reproducer. ```python import aiohttp import fsspec url =...