argo-events icon indicating copy to clipboard operation
argo-events copied to clipboard

don't trigger more if already too many workflows pending

Open tooptoop4 opened this issue 1 year ago • 0 comments

the idea is to prevent https://github.com/argoproj/argo-workflows/issues/10441

ratelimit on sensor is not enough to prevent this because slowing the creation of workflows does not guarantee they will complete quickly, so given a low parallelism of how many actually run and finish the number of pending workflows can grow too high.

my proposal is this: at startup of the sensor get the number of pending workflows in argo, if greater than x then make the sensor abort, also periodically check the number of pending workflows in argo every Y seconds, if greater than x then make the sensor abort

tooptoop4 avatar Feb 02 '23 20:02 tooptoop4