zarr-python
zarr-python copied to clipboard
[WIP] Boundary chunk optimization and typing
This PR addresses #757. is_total_slice takes a slice-like argument and a shape and returns True if the slice-like argument "fills" the shape, False otherwise. When performing writes, the output of is_total_slice determines whether a chunk is completely re-written (fast) or read -> modified -> written (slower). In master, is_total_slice is always called with the array's chunk shape as the second argument, and thus this function always returns False for chunks at the boundary of an array. This PR adds a new function zarr.util.trim_chunks in that essentially computes effective_chunk_shape = (chunk_shape * num_chunks) - array_shape and passes this value to is_total_slice, which allows writing to boundary chunks without a read operation.
Although all the core tests are passing, there's a deadlock in the test_parallel_append test in the test_sync suite. I have no idea what could be causing this.
Additionally, I kind of went crazy with adding type hints. It's possible that I will use this PR to add type hints to the entire library. It's possible that I should split the typing stuff into its own PR, but since none of it affects runtime I figured it couldn't hurt to combine it with some functional changes.
TODO:
- [x] Add unit tests and/or doctests in docstrings
- [ ] Add docstrings and API docs for any new/modified user-facing classes and functions
- [ ] New/modified features documented in docs/tutorial.rst
- [ ] Changes documented in docs/release.rst
- [ ] GitHub Actions have all passed
- [ ] Test coverage is 100% (Codecov passes)
Hello @d-v-b! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:
- In the file
zarr/core.py:
Line 1645:101: E501 line too long (129 > 100 characters)
Comment last updated at 2021-11-08 00:14:08 UTC
:+1: for the addition of type annotations from my side.
Davis mentioned there is a process synchronization test failing. We discussed briefly how to debug this a bit further to see what is going on there