apache
[DSIP-98][Worker] Add per-task-type maximum concurrency limits on Workers
Search before asking
- [x] I had searched in the DSIP and found no similar DSIP.
Motivation
todo:
Design Detail
todo:
Compatibility, Deprecation, and Migration Plan
No response
Test Plan
No response
Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Can you confirm if my understanding is correct? For example, suppose we have worker1 and worker2, and worker1 has more resources (such as CPU and memory) compared to worker2.
In such an unbalanced cluster environment, I can set different concurrency limits for worker1 and worker2. For instance, on worker1, the concurrency for shell tasks can be set to 10, and for SQL tasks, it can be set to 5. Meanwhile, since worker2 has fewer resources, the concurrency for shell tasks on worker2 can be set to 5, and for SQL tasks, it can be set to 2.
Can you confirm if my understanding is correct? For example, suppose we have worker1 and worker2, and worker1 has more resources (such as CPU and memory) compared to worker2.
In such an unbalanced cluster environment, I can set different concurrency limits for worker1 and worker2. For instance, on worker1, the concurrency for shell tasks can be set to 10, and for SQL tasks, it can be set to 5. Meanwhile, since worker2 has fewer resources, the concurrency for shell tasks on worker2 can be set to 5, and for SQL tasks, it can be set to 2.
Yes, this is a use case. However, our worker node configurations are typically identical. We aim to support varying levels of concurrency for different tasks—for instance, SQL/HTTP tasks can handle higher concurrency, while shell tasks support lower concurrency.
I think this feature is great. Our tasks are mostly Seatunnel + SQL, and we need to control Seatunnel concurrency without limiting SQL concurrency.
TaskGroup can also be used to control the concurrency of task. This new feature may have some conflict with the existing TaskGroup
TaskGroup can also be used to control the concurrency of task. This new feature may have some conflict with the existing TaskGroup
It can be extended based on the existing TaskGroup functionality.The existing TaskGroup functionality is tied to the project, and I don't understand why it's designed this way. I think it's a bad design.
TaskGroup can also be used to control the concurrency of task. This new feature may have some conflict with the existing TaskGroup
TaskGroup feels rather peculiar to use. In most cases, we use worker groups to partition resources and manage concurrency. It is difficult to enforce users to set TaskGroups for tasks, but tasks must utilize worker groups and workers.