loky icon indicating copy to clipboard operation
loky copied to clipboard

cpu_count_physical fails on some platforms / potential better default

Open surfaceowl opened this issue 5 months ago • 1 comments

Issue: The loky context manager relies only on the lscpu util to get physical and logical cpu for the _count_physical_cores method, and raises a ValueError of found {cpu_count_physical} physical cores < 1` when lscpu is missing or fails. This makes loky less cross-platform friendly than it could be.

Impact: On systems where lscpu is missing or fails, a ValueError is raised as the string found 0 physical cores < 1 trigging a failure when creating multiprocessing Pools on these systems. This happens both when the env var LOKY_MAX_CPU_COUNT is set and when it is missing or unset.

Example containers where it is missing include Heroku and Alpine. Docker containers your team is building are easy to fix, but some platform providers make it more difficult (cough, cough Heroku), and would be nice to reduce Docker build noise for missing libraries.

Suggested solutions for consideration:

  1. replace current ValueError with a try/except of other backup methods for lscpu that are more cross-platform friendly. This could be methods like os.cpu_count(), or cat proc/cpuinfo or some other *nix utility. This may have some downsides on accuracy on logical vs. physical cores, or cores not actually available to the process I'm not aware of... and I am not fully conversant on which utils are available on what platforms.
  2. _count_physical_cores should return an INT value of 1 + a warning, not a string and ValueError. If any python is run then we have at least 1 CPU, and can emit a loud warning that _count_physical_cores did not work correctly. A users multiprocessing pool would surely be slow, but it would at least run rather than fail. Picking the wrong number of cpu would slow down multiprocessing, but that may be preferred to not running at all.
  3. consider fancy checking availability for a small number of utils and using what is installed e.g. lscpu vs nproc or even psutil - though I understand the project does not want psutil as a hard dep from tests.

Additional Context: Other commands exist may give sufficiently accurate CPU info and be more broadly cross-platform:

  • os.cpu_count() which comes with python;
  • reading cat /proc/cpuinfo and splitting strings to get siblings for number of logical CPU and cpu cores` for number of physical CPU.
  • other linux utils like nproc though I don't know how many of these are out there, and given the number of different platforms out there it may not be possible to find the silver bullet.
  • Finally there is also the python lib psutil.

Separately - on Heroku I was able to install lscpu but due to some backend magic by Heroku maintainers, lscpu still failed from the cli returning the error lscpu: failed to determine number of CPUs: /sys/devices/system/cpu/possible: No such file or **directory.**

In discussion with Heroku maintainers, they suggested using 'nproc': """ ...just to follow up on my colleague's response: on Linux, please use nproc (available on Heroku's base images, from GNU coreutils) to determine the number if available processor cores.

Unlike lscpu, nproc will correctly represent the number of CPU cores that are available to the current process after any cgroup restrictions are applied, which might be the case e.g. in a local Docker environment, and could be employed by Heroku in the future on shared dynos.

In contrast, lscpu will always report the number of online cores, even if a process may not have access to all of them.

Observe the following differences in output for a container restricted to CPU cores 1 and 2:

~ % docker run --rm -ti --cgroupns host --cpuset-cpus 1-2 heroku/heroku:22 bash

root@5d5abb8e0108:/# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit Byte Order: Little Endian CPU(s): 6 On-line CPU(s) list: 0-5 Vendor ID: Apple Model: 0 Thread(s) per core: 1 Core(s) per socket: 6 Socket(s): 1 …

root@5d5abb8e0108:/# nproc 2

root@5d5abb8e0108:/# nproc --all 6 """

surfaceowl avatar Feb 20 '24 21:02 surfaceowl