pipeline
pipeline copied to clipboard
tektoncd
Ability to throttle concurrent TaskRuns
Feature request
I would like the ability to throttle a certain Task to have a maximum number of concurrent TaskRun at a time. IE...if I submit a task 100 times, and I have a throttle set to 5, the tasks would complete with a maximum of 5 running at a time (with the backlog queuing once another slot opens). This might not need to be a tekton specific functionality, but I have tried to implement this with standard kubernetes ResourceQuotas and Object Count Quotas, but it doesn't take into account the "running" state of the task. I could probably get something working that's non-precise with regular cpu/memory limits as well, but it would be nice to control this precisely. If there's another workaround approach, I'm all ears.
Use case
We have a task in a pipeline that
- has an expensive API call with strict rate limit rules (to an external service which I don't control)
- can be submitted in bursts where a lot can be running concurrently.
My goal is prevent rate limit exceptions with the external service.
Thank you!
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.
/lifecycle stale
Send feedback to tektoncd/plumbing.
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.
/lifecycle rotten
Send feedback to tektoncd/plumbing.
TEP-0120 is working on canceling concurrent pipeline runs - it does not support queuing but that could be another concurrency strategy we add
We do not handle this today but it's an interesting use case for the future, so marking as frozen
