pipelines icon indicating copy to clipboard operation
pipelines copied to clipboard

Published 20 hours ago • verified publisher icon kubeflow

[feature] Posibility to set tolerations for driver pods

Open jan-stanek opened this issue 1 year ago • 2 comments

Feature Area

/area backend

What feature would you like to see?

It would be great to have posibility to set tolerations also for container-driver and dag-driver.

What is the use case or pain point?

We have a cluster where all nodes are tainted, so we are not able to execute any pipeline.

Is there a workaround currently?

It is not possible, we have to use v1 pipelines.

Love this idea? Give it a 👍.

jan-stanek avatar Oct 18 '24 07:10 jan-stanek

The interface for controlling driver is typically environment variables. Would you want to set driver tolerations in pipeline code, or would you be ok with an env var on the apiserver deployment that applied to all runs across all pipelines? I ask because from a user api perspective, I don't love leaking things about driver into pipeline code.

gregsheremeta avatar Oct 18 '24 21:10 gregsheremeta

Env var on the apiserver deployment is enough

jan-stanek avatar Oct 19 '24 19:10 jan-stanek

@gregsheremeta do you know if it can be added in this project or argo workflows has to be changed too?

jan-stanek avatar Nov 21 '24 07:11 jan-stanek

There would be no need to make any Argo Workflows modifications. It's probably as simple as modifying this template, but I didn't look too closely.

gregsheremeta avatar Nov 25 '24 22:11 gregsheremeta

Do you have any idea how to store tolerations specification into env variable? From this perspective tolerations set in pipeline code would be better.

jan-stanek avatar Dec 13 '24 14:12 jan-stanek

I personally dislike the idea of leaking anything about driver (which is a backend implementation detail) into the user-facing API, which is why I asked if you'd be ok with it going in an env var.

Do you have any idea how to store tolerations specification into env variable?

I'd probably use a generic json string.

gregsheremeta avatar Dec 13 '24 15:12 gregsheremeta

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Feb 12 '25 07:02 github-actions[bot]

It is still needed

jan-stanek avatar Feb 12 '25 07:02 jan-stanek

Any updates on this? I'm currently working in an environment where each pod requires tolerations, so I'm unable to run any pipelines at the moment.

luko0610 avatar Mar 24 '25 11:03 luko0610

I had no time to implement it yet, but we will still need it in near future.

jan-stanek avatar Mar 24 '25 11:03 jan-stanek