dfdx
dfdx copied to clipboard
Safetensors changes
- Makes the safetensors module private.
- Doesn't get exported on the preamble, avoiding a naming clash with the safetensors external crate for apps that use dfdx.
- Change how and when the period
.
is inserted.- This should make it closer to how the fields are accessed in the code.
- For load, read, save, write safetensor(s), add a
_with
method:- Requires load/read to decide whether it should skip missing tensors;
- Requires load/read/save/write to decide how should keys be mapped.
- This makes it easier to load/save from/to models that were not saved by dfdx and thus contains a different safetensors key/location structuring.
- Allow models to get built from a safetensors byte array data (not from reading a file).
This doesn't have much testing nor documentation updates, so this PR should be considered a draft.