orbax
orbax copied to clipboard
HNS checkpoint folder moved to _todelete_subdirectory for future deletion to avoid slow listing
When a checkpoint folder is renamed, it's moved into a todelete subdirectory. This directory is created as a sibling to the main checkpoints/ folder.
This approach prevents these renamed folders from being listed during a checkpoint restore operation, which lists the contents of the checkpoints/ directory.
Note: This implementation uses a direct GCS API call as a temporary solution. This functionality will be migrated to tf.io once it becomes available there.