flaxmodels
flaxmodels copied to clipboard
Use `safetensors` to store tensors instead of `pickle`
Hi @matthias-wright, I've been playing around for a couple days with your project and it's so cool, thanks for building some pure flax
models here 👍🏻
Don't know if you're aware, but @huggingface developed a new format for storing tensors named safetensors
as most of the serialized models from PyTorch use pickle
to store the tensors, which seems to be not super efficient plus it has some known security issues. So I want to know whether you're considering to port the current tensors to use safetensors
instead.
I've recently built safejax
so as to easily do that, which means that the storage is optimal and more safe! If this is something you could consider to improve flaxmodels
please let me know and I can try to help if applicable!
P.S. Did you consider publishing the Python package to PyPI tracking it through GitHub Release so that it attracts more users due to the ease of installation through pip
from PyPI instead of from source as in the README.md
?