[Bug] [Master server] Tasks in the same process are repeatedly delivered.

[q4q5q6qw]

Search before asking

• [X] I had searched in the issues and found no similar issues.

What happened

First, create a shell workflow that contains only one shell script task. After the workflow runs properly, the monitoring information on the portal shows that the worker node breaks down. At this time, tasks in the workflow instance list are being delivered properly, but a new task instance is delivered for the tasks in the same workflow every second. After the worker node is restarted, the repeated tasks are executed concurrently. As a result, the CPU of the worker node explodes.

What you expected to happen

Tasks Are Not Repeatedly Delivered After the Worker Breaks Down

How to reproduce

Prepare a normal Dolphin cluster, create a workflow, and add a shell task to the workflow. After the scheduling is normal, stop the worker service process and check the number of times that the task instance is delivered.

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[wangxj3]

After the worker-server is down, should the backlog of task instances be run or not? If run it, it will cause excessive pressure on the worker-server, as you said. If it is not run, a large number of task statuses will be abnormal. But when running, you can reduce server pressure by configuring the number of concurrent threads in worker-server's exec-threads (default 100).

[q4q5q6qw]

After the worker-server is down, should the backlog of task instances be run or not? If run it, it will cause excessive pressure on the worker-server, as you said. If it is not run, a large number of task statuses will be abnormal. But when running, you can reduce server pressure by configuring the number of concurrent threads in worker-server's exec-threads (default 100).

worker should be executed, but the scene is a different case. I created a process, and only one task is created in this process. Normally, only one process_instance and a corresponding task_instance are generated. But now the situation is that a process_instance and multiple task_instances are generated, and before the worker is not restored, the task will be generated every 1 second.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.