Cannot use all CPUs at the same time

[versatile712]

General

• Operating System: Windows Server 2022 Datacenter
• Python version: 3.9.13
• Pandas version: 1.4.4
• Pandarallel version: 1.6.4

Acknowledgement

• [√] My issue is NOT present when using pandas without alone (without pandarallel)
• [√] If I am on Windows, I read the Troubleshooting page before writing a new bug report

Bug description

I have a server with 2 CPUs in 2 socks (64 physical cores, 128 threads in total). However, I can only use up to 32 nb_workers at a time, otherwise, my code will get stuck. The default parameter of nb_workers is 64 by pandarallel.

Observed behavior

The code gets stuck when using more than 1 CPU.

Expected behavior

Use all the cores in both socks.

Minimal but working code sample to ease bug fix for pandarallel team

Change the default parameter of nb_workers to the max cores of one single CPU, but it is more desired to use all the CPUs.

[nalepae]

Pandaral·lel is looking for a maintainer! If you are interested, please open an GitHub issue.

[shermansiu]

"The code gets stuck when using more than 1 CPU."

I can't reproduce this, even when using psutil.cpu_count(logical=False) workers.

Do you have some minimally-working Pandas code that lags when you use all of your CPU cores?

[versatile712]

"The code gets stuck when using more than 1 CPU."

I can't reproduce this, even when using psutil.cpu_count(logical=False) workers.

Do you have some minimally-working Pandas code that lags when you use all of your CPU cores?

Hi, thanks for your help! I ran into this problem in one of my previous projects, and I couldn't locate the codes with problems now. I cannot reproduce the problem at the current moment (which means I can use all the cores now ORZ). It may be related to specific codes. When I run into the problem next time, I will paste the codes here.