kerchunk

kerchunk copied to clipboard

Refactor file format backend openers

1

### Problem The [API for Kerchunk's file format backend openers](https://fsspec.github.io/kerchunk/reference.html#file-format-backends) doesn't follow a consistent pattern. ### Suggestion Change the openers to each be a function returning a `VirtualZarrStore` (see #375),...

TomNicholas

Concatenating datasets with staggered grids

6

Here are two toy datasets designed to represent sections of a dataset that has variables living on a staggered grid. This type of dataset is common in fluid modelling (handling...

TomNicholas

Represent (coordinate) variables "symbolically"

8

I'm working with a GRIB2 file, and am interested in minimizing the size of the references file. Currently, the largest values in the references come from the base64-encoded coordinates that...

TomAugspurger

Create coordinate from dimension?

1

Hi! I've been trying to convert an existing dimension in my ROMS model output file (e.g. eta_rho) into a coordinate when generating the `MultiZarr` output. This would allow me to...

kthyng

building an index for *remote* zarr sources fails, due to `single_zarr` bug

4

Was hoping to use `single_zarr` to build an index for multiple remote zarr sources, for benchmarks against remote netcdf files, but it seems it's a test helper for combining local...

okz

Creating index for single zarr-in-zip stores

2

Picking up from https://github.com/fsspec/kerchunk/issues/220#issuecomment-1677349599 it would be great to have an example of creating a kerchunk index for a zarr store archived into a zip file, under the following assumptions:...

pbranson

Issue using `tiff_to_zarr`

10

Hi there @martindurant, I've started playing around with Kerchunk's `tiff_to_zarr` functionality and ran into an issue when trying to open up the reference file. In short, the `tiff_to_zarr` works successfully,...

norlandrhagen

`tiff_to_zarr` ValueError: incomplete chunks are not supported by the fsspec ReferenceFileSystem

4

Hi there, I'm working through generating some Kerchunk reference files for a set of GeoTIFFs and I'm getting the error: ```python ValueError: incomplete chunks are not supported by the fsspec...

norlandrhagen

Most of the scanners use an internal `out = {}` dict to accumulate references. This would be a writable parquet store, flushing blocks of references as they fill. This idea...

martindurant

TIFF: internal codec (small chunks) vs. entire file as single chunk (`imagecodecs_tiff` codec)

10

I just read #78 in detail. In that issue, @cgohlke and others noticed that many tiff files contain tiny internal chunks, and mapping these 1:1 to Zarr chunks is not...

rabernat