zarr-developers
Add safetensors reader
⚠️ DO NOT MERGE: ⚠️ This PR should not be merged until support for dtype extensions is added to zarr-python.
Summary
This PR is the first step in #367. It adds a reader for the official Safetensors specification (see specs). Note that HuggingFace currently implements a SafeTensor reader that goes considerably beyond the official specification. Support for this implementation (which in practice the entire ML/AI community relies on) will be added in the next step for #367.
Notes for reviewers
-
This PR was developed in association with an AI coding assistant. Make of that what you will!
-
I have added a notebook to show how everything works. This can be removed in the final version. I have a question at the end of the notebook related to partial get support, which I would appreciate a core dev looking into.
Checklist
- [x] Tests added
- [x] Tests passing
- [x] Full type hint coverage
- [ ] Changes are documented in
docs/releases.rst - [ ] New functions/methods are listed in
api.rst - [x] New functionality has documentation
