recommenders-addons
recommenders-addons copied to clipboard
[Feat]Copy-free save and load for cuckoo hashtable
Description
Brief Description of the PR: Since dynamic embedding could be super large for memory limit. save and load with traditional TensorFlow checkpoint mechanism will use a lot of memory when saving or loading. This PR provides a method to save or load files for dynamic embedding tables, without full volume copying.
Type of change
- [x] Bug fix
- [ ] New Tutorial
- [ ] Updated or additional documentation
- [x] Additional Testing
- [x] New Feature
Checklist:
- [x] I've properly formatted my code according to the guidelines
- [ ] By running yapf
- [ ] By running clang-format
- [ ] This PR addresses an already submitted issue for TensorFlow Recommenders-Addons
- [ ] I have made corresponding changes to the documentation
- [x] I have added tests that prove my fix is effective or that my feature works
How Has This Been Tested?
Yes
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
For more information, open the CLA check for this pull request.
This feature is very useful. Looking forward to it.
May I ask how to use it? In our case, we will use estimator and save checkpoints per epoch. So should we customize a saver to save the tables manually? But if so, how can we get all the tables to save?
Currently it only support usage like save_op = table.save(path)
. In eager mode, it's pythonic and simple. In graph mode, it should be managed on graph, like SessionRunHook or run sub-branch of graph.
This is a good solution, we want to solve this problem too, When can it be merged?
Try this: https://github.com/tensorflow/recommenders-addons/blob/master/docs/api_docs/tfra/dynamic_embedding/FileSystemSaver.md