Lachlan Deakin
Lachlan Deakin
Below are some benchmarks on my system. The memory usage of `zarrs-python` is curious. ### Read all  ### Chunk by chunk  | | Image | Concurrency | Time...
> Something that pops out to me - any thoughts on why the sharding might be so performant across teh board for the compiled stuff? There are two areas where...
Also looking at that benchmark again, do you have an idea of where the large allocation (8.7GB) is occurring in `zarrs-python` when concurrent chunks > 1?
> it would be preferable to reduce the concurrent target to avoid potential thrashing Quoting myself... but thrashing is not the right term here. That does not really happen with...
Cool stuff @kylebarron! Yeah, I'd be interested to see an integration
> Is there any interest? Yea! At the Zarr summit, there was a lot of interest to support more stores in `zarrs-python`. > Any rough outline of what would need...
This shouldn't be too hard on the `zarrs-python` side: - Convert Python variable length arrays to the `zarrs` representation [`ArrayBytes::Variable`](https://docs.rs/zarrs/latest/zarrs/array/enum.ArrayBytes.html) - Search for `into_fixed()`, `new_flen()` and add variable length support...
> Python bindings already exist so I think we would need rust bindings as well Unfortunately, no kvikio Rust bindings exist. I tried a few months ago and bailed. Happy...
> Is there a reason why you require Hash, PartialEq, Eq, PartialOrd, Ord on StoreConfig? Those are a lot of bounds. I think it is related to storing the configs...
> Crucially, we'd need a Zarr library that could efficiently load (at least) thousands of chunks per second. `Zarrs` feels like a great candidate to be able to do this....