flyte
flyte copied to clipboard
[Core feature] [flytekit] Support for Async checkpoints
Motivation: Why do you think this is important?
With https://github.com/flyteorg/flyte/issues/634, Flyte has the capability of checkpointing. Flytekit specifically in this PR https://github.com/flyteorg/flytekit/pull/771 implemented Synchronous checkpoints. But, synchronous checkpoints for extremely large checkpoints could be very expensive as they persist the checkpoint to a durable store synchronously. It is beneficial to checkpoint asynchronously in such cases. Care has to be taken that asynchronous checkpoints to do corrupt previous ongoing checkpoints and hence need some sort of serialization primitives.
Goal: What should the final outcome look like, ideally?
User should be able to async checkpointing using a very similar api.
Describe alternatives you've considered
NA
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
- [X] Yes
Have you read the Code of Conduct?
- [X] Yes