argo-workflows icon indicating copy to clipboard operation
argo-workflows copied to clipboard

feat: Artifact GC

Open juliev0 opened this issue 3 years ago • 1 comments

Fixes #8476

Note: may want to hold off on merging this until Dillen Padhiar's PR is merged which executes the logic in the Garbage Collection Pod (argoexec artifact delete). Although, his first draft code is currently contained within this branch, which is functional, if we choose to merge this first.

I have tested this under the following conditions:

  • different combinations of specifying ArtifactGC (Strategy, ServiceAccount, PodMetadata) on the Workflow level and Artifact level
  • ArtifactGC failure
  • Workflows based on StoredTemplates
  • Workflows whose steps reference templates in StoredTemplates
  • Workflows that use Steps and Workflows that use DAG

The following is not being implemented as part of this MVP and can be done later:

  • OnWorkflowSuccess,OnWorkflowFailure (with deletion)
  • Validation: Jesse suggested we reject the Workflow during validation if ArtifactGC is configured along with a non-supported storage engine (for now probably anything besides S3).
  • Creating multiple ArtifactGCTasks in the case that we start to exceed 1MB
  • handling recreation of the GC Pod in case it's evicted or deleted

juliev0 avatar Jul 29 '22 06:07 juliev0

I think you should descope the OnWorkflowFailure and OnWorkflowSuccess, validate the use case and if valid, follow up with another PR.

I removed those strategies and can create a new Issue. "OnWorkflowSuccessOrDeletion" and "OnWorkflowFailureOrDeletion" may make sense.

juliev0 avatar Aug 03 '22 18:08 juliev0