zarr-developers
Treatment of final smaller chunk in the zarr model
The ManifestArray code right now might not be treating carefully enough the possibility of the last chunk being smaller than the rest. For example if you concatenate two arrays that each have uniform chunks except for the final chunk, you could end up with an array that has variable-length chunks in the middle somewhere 😕
One solution to this might be to generalise ManifestArray.chunks to be able to represent rectilinear arrays of chunks (a step towards #33), but just have a check on the constructor to forbid creating an array that actually has any chunk other than the last be of different length. This check could then be relaxed in future once we have variable-length chunks supported upstream in zarr.
We need some test cases to start explicitly checking behaviour for these cases. Unfortunately kerchunk doesn't seem to have any (see https://github.com/fsspec/kerchunk/issues/436). This is a little worrying because this seems like an important set of edge cases to check, so we should try to find / create some files like this ourselves and add them to the test suite of this library.
have a check on the constructor to forbid creating an array that actually has any chunk other than the last be of different length.
A better place to have this check would probably be in the serialization step. It makes perfect sense to have a ManifestArray that points to variable-length chunks, the problem is just that neither of the available on-disk formats (kerchunk or zarr) support variable-length chunks yet. (A warning instead of an error in the constructor would be more appropriate.)