zarrs-python icon indicating copy to clipboard operation
zarrs-python copied to clipboard

Fix unsound assumption of chunks being disjoint

Open flying-sheep opened this issue 10 months ago • 1 comments

See https://github.com/ilan-gold/zarrs-python/pull/87#discussion_r1966169606

I assume we could still do things in parallel if we want.

We could create a DisjointViews (or so) struct that holds non-overlapping ArrayBytesFixedDisjointViews and has a safe fn add(&mut self, ...) -> bool that returns if a region was safely added (would not overlap with the already contained regions) or not. Then at runtime,

  1. create a Vec<DisjointViews> by first trying to add each view to the first DisjointViews object, then the second, … until each view has been added.
  2. go over the vec and handle each DisjointViews object in parallel.

There could be pathological cases where that doesn’t work well, but in practice, those shouldn’t happen

flying-sheep avatar Feb 24 '25 09:02 flying-sheep

That sounds reasonable, but we could just be patient. zarr-python might disallow it https://github.com/zarr-developers/zarr-python/issues/2855

LDeakin avatar Feb 24 '25 09:02 LDeakin