conductor-oss
Http Task not Picked soon
Describe the bug We are executing 100 Workflows per second in 4 Containers pods with 4 core in each pod. We observe that Http Task is not picked immediately. Updated activeWorkerLastPollTimeout to 2 sec where we saw improvements but not great improvement. Wonder is there any other setting we are missing.
In the above example Task2 is picked after 9 seconds and hence the join took more time.
Details Conductor version: v3.21.5 Persistence implementation: Cassandra, Postgres, MySQL, Dynomite etc Postgres Queue implementation: Postgres, MySQL, Dynoqueues etc Postgres Lock: Redis or Zookeeper? Workflow definition: Task definition: Event handler definition:
To Reproduce Steps to reproduce the behavior:
- Go to '...'
- Click on '....'
- Scroll down to '....'
- See error
Expected behavior Http Task pulled immediately and better metrics on pending Http Task Qeueu
Screenshots If applicable, add screenshots to help explain your problem.
Additional context We are comparing this with AWS Step Functions and this issue is holding us to move forward with Conductor
Hi All, Any help here is appreciated. Will task executionNamespace help to ensure priority tasks are picked? Our alternative is Cloud Step Functions but really wanted to see if Conductor can help.