terraform-provider-iterative icon indicating copy to clipboard operation
terraform-provider-iterative copied to clipboard

Support setting completions to a different value than parallelism for k8s tasks

Open sjawhar opened this issue 1 year ago • 3 comments

Requested Functionality

When running a large parallel job on a k8s cluster, I might have many more tasks that need to be run than I have pods to run them. In that case, it would be very useful to use k8s indexed jobs. A key part of this is the ability to set parallelism and completions to different values—specifically, parallelism would have a lower value than completions. See the discussion in https://github.com/iterative/terraform-provider-iterative/pull/597 for background and context.

Example TF file:

resource "iterative_task" "example" {
  cloud     = "k8s"
  machine   = "1-1024"
  image     = "python:3.8.12"
  disk_size = 1

  parallelism = 2
  completions = 5

sjawhar avatar Sep 05 '22 23:09 sjawhar

Related to #585, useful but harder to implement in backends other than k8s

0x2b3bfa0 avatar Sep 06 '22 02:09 0x2b3bfa0

Related to #585, useful but harder to implement in backends other than k8s

@0x2b3bfa0 let's start with k8s support only then

omesser avatar Sep 06 '22 08:09 omesser

You can see here how I implemented it. Should I open a PR, or would you prefer a different implementation?

sjawhar avatar Sep 09 '22 03:09 sjawhar