flyte
flyte copied to clipboard
[Core Feature]Automatic deduction of Cache Version for certain task types
Motivation: Why do you think this is important? Flytekit today relies on the Cache Version to be specified always. This can be error prone. But, in some cases deducing the Cache Version is completely possible. For example in the case of SQL Queries or Instance tasks, or sagemaker built-in algorithm tasks etc. Flytekit should allow some task types to automatically deduce the cache version
Goal: What should the final outcome look like, ideally? Users should not need to specify cache version of certain task types. It should be automatically de-ducted. But, cacheable should be optional.
Describe alternatives you've considered Implement the cache version on the user side. This is complicated and more error prone