kerchunk
kerchunk copied to clipboard
fsspec
Cloud-friendly access to archival data
I'm trying to understand fsspec-reference-maker better. Consider the following code from the pangeo-forge [hdf-reference tutorial](https://pangeo-forge.readthedocs.io/en/latest/tutorials/hdf_reference/reference_cmip6.html#manual-loading): ```python m = fsspec.get_mapper( "reference://", fo=ref_url, target_protocol="file", remote_protocol="s3", skip_instance_cache=True, ) ``` Please help me understand...
@jbusecke and I are wondering if https://github.com/intake/fsspec-reference-maker/blob/89d32322fdd3ba631a4fe846475a8857f10ad872/fsspec_reference_maker/combine.py#L313-L316 should instead check that `v / ds0.dims[k] > 1`. We were working with the Pangeo Forge [`HDFReference`](https://github.com/pangeo-forge/pangeo-forge-recipes/blob/c4a4ed84027ee6d2c372d8f403a3c3f3109a13d1/pangeo_forge_recipes/recipes/reference_hdf_zarr.py#L25) recipe class today, and it seemed...
Not so much an issue, but just some examples of applying this to a large 30 year Sea Surface Temperature (SST) dataset (10576 x 4500 x 6000), chunked in (1,...
As @martindurant mentioned in #58 we should probably get some tests created with the `example_*` functions and run them automatically with new PRs. I think this can be relatively easy...
The following formats can currently be processed: - HDF5 - tiff - grib2 Please comment about other file formats that might be interesting to this project. Have they been translated...
I am trying to create references for this [netCDF file](https://ted-habermann.squarespace.com/s/coloradoStations.nc) which is a collection of ~30 xArray dataSets. Each dataSet has 8 HDF5 datasets. This file was written with xArray....
Opening an issue so we can discuss processing time for MultiZarrToZarr. I'm running with the following code: ```python mzz = MultiZarrToZarr( json_list, remote_protocol='az', remote_options={ 'account_name' : 'goeseuwest' }, xarray_open_kwargs={ 'decode_cf'...
All datasets with fletcher32 == True are rejected by referenceMaker. This is the checksum supported by HDF5. Would it be possible to add it to nomcodecs? @ajelenak
https://medium.com/pangeo/cloud-performant-netcdf4-hdf5-with-zarr-fsspec-and-intake-3d3a3e7cb935 This blog post is great! There is huge potential in using unmodified HDF files in object storage and obtaining better read efficiency with fsspec and sidecar metadata. Often existing...
Two key things we may wish to implement: - building references on-the-fly, such that the first access to a dataset is slow, but then subsequent accesses use a cached references...