The continuous increase of workerserver processes, is this normal?

[peak-xu]

Search before asking

• [X] I had searched in the issues and found no similar issues.

What happened

exccute commond ps -eLf | grep WorkerServer | wc -l,result is increase,until the service is down

What you expected to happen

I dont know what the reason is, Perhaps we have a lot of scheduled tasks.

How to reproduce

standalone env and a lot of scheduled tasks,you will see the workerServer process is increase

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[ruanwenjun]

What kind of task, this might be sub process created by the worker.

[peak-xu]

What kind of task, this might be sub process created by the worker.

Scheduled Task ,Why does the number of processes continue to increase? My understanding is that once a process finishes running, it should disappear.

[ruanwenjun]

Scheduled Task ,Why does the number of processes continue to increase? My understanding is that once a process finishes running, it should disappear.

pf -eLf will print the thread in the process, since we use a fixed thread pool, so the thread num will increase until exceed exec-threads then it will not increase, and it will not release.

[peak-xu]

Scheduled Task ,Why does the number of processes continue to increase? My understanding is that once a process finishes running, it should disappear.

pf -eLf will print the thread in the process, since we use a fixed thread pool, so the thread num will increase until exceed exec-threads then it will not increase, and it will not release.

I found that a task occupied more than 7,000 threads.

[wangxj3]

check the path of /proc/${worker-server pid}/task/ ,you can get the detail about the sub threads of the worker-server.Check the task to see if there is a scheduled thread pool or other logic for creating threads. The thread pool is still creating threads regularly after the daemon thread ends.Hope it helps

[peak-xu]

check the path of /proc/${worker-server pid}/task/ ,you can get the detail about the sub threads of the worker-server.Check the task to see if there is a scheduled thread pool or other logic for creating threads. The thread pool is still creating threads regularly after the daemon thread ends.Hope it helps

I don’t know if I used ps -eLf | grep workerServer to count the number of threads correctly. It does not match /proc/pid/task. By observing the results of the ps -eLf command, I guess that some task threads are not destroyed after execution. Once the number of threads reaches a certain If the quantity is too high, the service will be down. I don’t know how to analyze it next.

[ruanwenjun]

check the path of /proc/${worker-server pid}/task/ ,you can get the detail about the sub threads of the worker-server.Check the task to see if there is a scheduled thread pool or other logic for creating threads. The thread pool is still creating threads regularly after the daemon thread ends.Hope it helps

I don’t know if I used ps -eLf | grep workerServer to count the number of threads correctly. It does not match /proc/pid/task. By observing the results of the ps -eLf command, I guess that some task threads are not destroyed after execution. Once the number of threads reaches a certain If the quantity is too high, the service will be down. I don’t know how to analyze it next.

Can you print the thread name?

[peak-xu]

[peak-xu]

Is the thread status sleeping normal?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

[czxin788]

This is my dolphinscheduler workserver thread nums monitor:

ps -eLf | grep 'WorkerServer' | wc -l

and the result is :

When the value reaches 20000+, the all task in dolphinscheduler will hung,all task alway in running state and not end up.

Are there a solution methods?