zarr-python icon indicating copy to clipboard operation
zarr-python copied to clipboard
verified publisher icon zarr-developers

feat/global concurrency limit

Open d-v-b opened this issue 2 months ago • 1 comments

Makes the concurrency limit global, rather than per-function.

I'm not emotionally attached to this PR -- it's largely the work of claude code. Lets use this as a starting point for improving on #3526

d-v-b avatar Oct 24 '25 19:10 d-v-b

Nice, the other one that bugs me when profiling is https://github.com/zarr-developers/zarr-python/blob/b3e9aed305092236c5db70deee0b26dad648d3b0/src/zarr/codecs/zstd.py#L79

dcherian avatar Oct 27 '25 21:10 dcherian

@d-v-b do you have thoughts on how downstream libraries would interact with Zarr's concurrency limit? Would those set there own concurrency approaches (e.g., global or per function) or could there be a way to share the limit with Zarr?

maxrjones avatar Dec 20 '25 22:12 maxrjones

@maxrjones this PR is moving away from a global concurrency limit because it's actually very hard to implement a top-down concurrency control in our codebase. the new direction I'm taking with this PR is for each store to set its own concurrency limit. for stateless stores like local and remote (basically anything except zip), concurrency-sensitive applications should be able to cheaply create store instances with concurrency limits tuned to the needs of the application.

d-v-b avatar Dec 22 '25 10:12 d-v-b