digdag icon indicating copy to clipboard operation
digdag copied to clipboard

[feature request] Add worker agent information to task queue

Open TrsNium opened this issue 4 years ago • 0 comments

I am trying to run digdag on kubernetes. Most functions work properly on kubernetes.

However, it is difficult to handle the scale-out and scale-in behavior of the digdag worker agent. If scale out operation exceeds max-task-threads of all worker agents in task queue, scale out must be performed. However, the scale-in operation cannot be performed unless the task queue is empty. Because we don't know which worker agent is processing the task. This crd of kubernetes(https://github.com/TrsNium/digdag-worker-crd) is the above scale-in and scale-out behavior.

Therefore, we want to add information on which worker agent is processing the task in the task queue and make the scale-in and scale-out processing flexible.

TrsNium avatar Mar 01 '20 08:03 TrsNium