a-guide-to-mlops icon indicating copy to clipboard operation
a-guide-to-mlops copied to clipboard

Do not create GPU pod when running on main branch

Open rmarquis opened this issue 1 year ago • 0 comments

Currently, the GPU pod is also created when pull request is merged into the main branch, before being destroyed without the intermediary step being executed. To restrict the GPU pod creation to only when a pull request is opened, two considerations must be addressed:

  • The cleanup of the GPU pod should always be executed, even in the event of a failure, to prevent 'ghost' GPU pods from staying in the Kubernetes cluster.
  • No code should be pushed to the main branch without prior training (dvc repro and dvc push). Branch protection rules need to be set up to enforce this requirement. This particular issue is tracked in #199.

rmarquis avatar Oct 08 '24 08:10 rmarquis