mljet
mljet copied to clipboard
Replace default pickle serializer with joblib
We currently use Pickle as a serializer for machine learning models.
There are the following alternatives:
joblib- a serializer that is backward compatible with Pickle, but is more optimized for big data and also faster.dill- serializer, backward compatible with Pickle, which allows to serialize more objects, including lambda functions.
Based on our subject area, Joblib is the most suitable.
Steps to accomplish this task:
- [ ] Add
joblibrequirements to all backends templates. - [ ] Replace
picklecalls in backends server files. - [ ] Replace serialization in project builder.
- [ ] Update CLI commands.
- [ ] Check for other
pickleusages.
It is worth noting that we should support importing existing model dumps with different serializers, especially through the CLI interface. Another task will be created based on this.
СС: @pacifikus