argo-workflows
argo-workflows copied to clipboard
argoproj
Artifact Garbage Collection
Summary
When a Workflow is deleted, we should consider garbage collecting artifacts from those workflows where possible. This should likely be configurable.
Motivation
Right now, manual intervention is needed in order to clean up artifacts that were created as part of Workflows if desired. For high frequency workflows with high amounts of artifacts (think a CI system), cleaning up these artifacts can be pretty important for cost savings.
Proposal
Each type of artifact storage will need to support a deleteArtifact function. When a Workflow is deleted (not simply Archived, but actually deleted), all artifacts associated with that workflow would be cleaned up automatically.
We could make this configurable with something like
# The rest of the output config was omitted
outputs:
artifacts:
- name: my-output-artifact
artifactGC:
strategy: Never | OnWorkflowDeletion | OnWorkflowArchival
Initially, I'd suggest that the default be Never to match existing behavior unless you're OK with making a breaking change.
Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.
@pbebbo At the moment, it's on hold while I'm reconsidering how to actually implement this. When you consider the huge variety of ways that people can configure their artifact repo on AWS alone such as Access Key/Secret, IRSA, KIAM/kube2iam, temporary secrets that are cleaned up after the workflow runs, etc (and that's just AWS), the scope of the problem is actually pretty big if you try to infer what credentials you should use to delete any given artifact. All of that is actually assuming that you actually have delete permissions on your artifact repo using those credentials as well, and that's not necessarily true.
The current idea I've been kicking around is to add some additional config that would allow you to specify an additional set of credentials to use for garbage collection that are expected to be allowed to delete artifacts from any workflow which basically punts that responsibility to the user. With that in place, I think that should eliminate all of those concerns.
Note that my GCS deletion related work is present here if someone wants to work on top of that: https://github.com/argoproj/argo-workflows/pull/7659/files
For MVP, I think I'll just implement:
- S3 artifacts.
- Deletion on workflow deletion.
I'll omit:
- Non-S3, e.g. OSS and GCS (I don't have test env)
- Deletion on workflow completion.
Give this comment a 👍 if you want to specify once for the whole workflow. All artifacts use same GC.
I've completed a PoC (#8530) and @jessesuen and I have agreed on a design in that PoC.
I'm now asking for help with the following:
- We only currently support non-S3 storage. Could you help with implementing those?
- There are many issues in the artifacts project. There are in order with sizing and "good first issue" labels. Would you like to help there?
This is a great new feature but wondering if you'd considered more strategies eg a TTL for artifacts? This could be achieved by adding object expiration configuration when saving to blob storage.
Excuse me for my ignorance, Just trying to understand, wouldn't using a proper data streaming service as a means to transfer the artifacts (eg. Kafka and the like) would inherently solve this issue as the data obviously will be automatically deleted after being pulled?
