[Feature] Datacatalog cache deletion
Tracking issue
https://github.com/flyteorg/flyte/issues/2867
Why are the changes needed?
Flyte doesn't support the deletion of cached task executions. Adding cache eviction/cleanup gives Flyte users more control over their data helping with things such as GDPR compliance + reducing their cache size.
Opted to support just single task executions as invalidating an entire workflow's cache can run into timeout issues going down the nesting of nodes.
What changes were proposed in this pull request?
- add flyteadmin endpoint to support the deletion of single task executions
- add catalog endpoint to support the deletion of single task executions
How was this patch tested?
-
unit tests
-
Run workflow that caches task executions
grpcurl -plaintext -d '{"task_execution_id": {"node_execution_id": {"execution_id": {"project": "flytesnacks", "domain": "development", "name": "fbcf25e4303104b4b81e"}, "node_id": "n0"}, "task_id": {"project": "flytesnacks", "domain": "development", "name": "basics.cache_test.basic_cache", "version": "98XuDRSQW4hVWKYQdQfd1Q=="}, "retry_attempt": 0}}' localhost:8089 flyteidl.service.CacheService.EvictTaskExecutionCache
Setup process
- if want to verify from console, need to pull down: https://github.com/flyteorg/flyteconsole/pull/835 and then manually update flyteconsole/packages/flyteidl-types/node_modules/@flyteorg/flyteidl/gen/pb-js/flyteidl.d.ts
enum CatalogCacheStatus {
CACHE_DISABLED = 0,
CACHE_MISS = 1,
CACHE_HIT = 2,
CACHE_POPULATED = 3,
CACHE_LOOKUP_FAILURE = 4,
CACHE_PUT_FAILURE = 5,
CACHE_SKIPPED = 6,
CACHE_EVICTED = 7
}
(NOTE - a follow up PR will be to get that change merged into console since console doesn't support CACHE_EVICT CatalogCacheStatus)
- can also verify by checking artifact_data and artifacts tables
Screenshots
Check all the applicable boxes
- [x] I updated the documentation accordingly.
- [x] All new and existing tests passed.
- [x] All commits are signed-off.
Related PRs
Docs link
Codecov Report
Attention: Patch coverage is 67.56098% with 133 lines in your changes are missing coverage. Please review.
Project coverage is 59.07%. Comparing base (
e07084c) to head (9e322a8).
Additional details and impacted files
@@ Coverage Diff @@
## master #4655 +/- ##
==========================================
+ Coverage 59.00% 59.07% +0.07%
==========================================
Files 645 647 +2
Lines 55578 55972 +394
==========================================
+ Hits 32792 33065 +273
- Misses 20194 20301 +107
- Partials 2592 2606 +14
| Flag | Coverage Δ | |
|---|---|---|
| unittests | 59.07% <67.56%> (+0.07%) |
:arrow_up: |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
put up change for console: https://github.com/flyteorg/flyteconsole/pull/851