kerchunk
kerchunk copied to clipboard
fsspec
Cloud-friendly access to archival data
Reposting from my email to @martindurant: > This fsspec-reference-maker, which indexes byte positions in HDF5 files and serves the HDF5 files as Zarr data—such a thing could be useful for...
I'm encountering an interesting issue where the results of scan_grib differ from interacting with a file via xarray/cfgrib. Particularly, it is not detecting certain variables. Installation information: ``` Python 3.11.0...
I'm trying to create a reference dict with inlined data, but the `inline_threshold` argument to `SingleHdf5ToZarr` doesn't seem to be doing anything. I feel like I'm missing something obvious. ```python...
Basically I want to check some assumptions I'm making about concatenation of kerchunk references in VirtualiZarr (for [this issue](https://github.com/TomNicholas/VirtualiZarr/issues/38)). I would like to know if when I scan e.g. a...
I'm looking for a combination of https://github.com/ProjectPythia/kerchunk-cookbook/blob/main/notebooks/advanced/Parquet_Reference_Storage.ipynb and https://github.com/ProjectPythia/kerchunk-cookbook/pull/59 and have `auto_dask` write to a parquet file
If https://github.com/ProjectPythia/kerchunk-cookbook/pull/59 gets merged it can be added in the doc string of https://github.com/fsspec/kerchunk/blob/main/kerchunk/combine.py#L768
Working on https://github.com/fsspec/kerchunk/issues/441
It seems [NCEP](https://github.com/NOAA-EMC/NCEPLIBS-grib_util/issues/293) add several custom encodings to the WMO Grib2 standard that the CFGRIB library can't decode. In the [HRRR SubHourly product](https://www.nco.ncep.noaa.gov/pmb/products/hrrr/hrrr.t00z.wrfsubhf02.grib2.shtml) the Step is encoded as a range...
Currently one can write to json outside of the function ``` zarrchunks = kerchunk.zarr.ZarrToZarr(uri) with open(out_json, "wb") as f: f.write(ujson.dumps(zarrchunks)) ``` Given this is a common operation it could be...
### Problem `MultiZarrToZarr` is extremely powerful but rather hard to use. **This is important** - kerchunk has been _transformative_, so we increasingly recommend it as the best way to ingest...