psutil
psutil copied to clipboard
cpu_count_physical() always returns None on arm devices
Similar to what was reported in #200 about arm systems not having the same /proc/cpuinfo format as x86 systems (and likely other arches too) and cpu_count_physical() for linux systems with arm will always return None.
We should either teach cpu_count_physical() about the different cpuinfo format, or in the alternative use /proc/stat like what is done in cpu_count_logical(). Arm doesn't have logical cores, so the parsing can be the same as cpu_count_logical() for an arm cpu.
Can you paste the output of cat /proc/cpuinfo
on ARM?
Sure, on the aarch64 system I'm running the /proc/cpuinfo contents:
Processor : AArch64 Processor rev 4 (aarch64)
processor : 0
model name : AArch64 Processor rev 4 (aarch64)
BogoMIPS : 26.00
BogoMIPS : 26.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
processor : 1
model name : AArch64 Processor rev 4 (aarch64)
BogoMIPS : 26.00
BogoMIPS : 26.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
processor : 2
model name : AArch64 Processor rev 4 (aarch64)
BogoMIPS : 26.00
BogoMIPS : 26.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
processor : 3
model name : AArch64 Processor rev 4 (aarch64)
BogoMIPS : 26.00
BogoMIPS : 26.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
Hardware : MT6797X
...so the right number to return in this case is supposed to be 4?
Yes I believe so, there were 4 active physical cores on that device when I read /proc/cpuinfo. (The CPU actually has more cores, but switches between them dynamically based on load and battery)
Can you paste the output of python -c "print(repr(open('/proc/cpuinfo').read()))"
?
Sure, no problem:
'Processor\t: AArch64 Processor rev 4 (aarch64)\nprocessor\t: 4\nmodel name\t: AArch64 Processor rev 4 (aarch64)\nBogoMIPS\t: 26.00\nBogoMIPS\t: 26.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nprocessor\t: 5\nmodel name\t: AArch64 Processor rev 4 (aarch64)\nBogoMIPS\t: 26.00\nBogoMIPS\t: 26.00\nFeatures\t: fp asimd evtstrm aes pmull sha1 sha2 crc32\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nHardware\t: MT6797X\n'
I've confirmed this is the case on an AArch64/arm64 Linux 4.15 (not Android) virtual machine to which I have access as well. The contents of /proc/cpuinfo indicate 8 physical processors and psutil.cpu_count() returns 8 but psutil.cpu_count(logical=False) returns None.
So, this is a patch which relies on lines such as processor: 1
and assume they are physical cores. I'm gonna put it here and do not commit this yet because I am not 100% convinced this is the right approach.
diff --git a/psutil/_pslinux.py b/psutil/_pslinux.py
index ecc4c703..faf954cb 100644
--- a/psutil/_pslinux.py
+++ b/psutil/_pslinux.py
@@ -642,8 +642,26 @@ def cpu_count_physical():
key, value = line.split(b'\t:', 1)
current_info[key] = int(value)
+ found = sum(mapping.values())
+ if not found:
+ # Perhaps an ARM system, see:
+ # https://github.com/giampaolo/psutil/issues/1359
+ found = set()
+ with open_binary('%s/cpuinfo' % get_procfs_path()) as f:
+ for line in f:
+ line = line.strip().lower()
+ if line.startswith(b'processor'):
+ key, value = line.split(b'\t:', 1)
+ try:
+ value = int(value)
+ except ValueError:
+ pass
+ else:
+ found.add(value)
+ found = len(found)
+
# mimic os.cpu_count()
- return sum(mapping.values()) or None
+ return found or None
def cpu_stats():
diff --git a/psutil/tests/test_linux.py b/psutil/tests/test_linux.py
index 4b72f725..042d1a7d 100755
--- a/psutil/tests/test_linux.py
+++ b/psutil/tests/test_linux.py
@@ -679,6 +679,44 @@ class TestSystemCPU(unittest.TestCase):
self.assertIsNone(psutil._pslinux.cpu_count_physical())
assert m.called
+ def test_cpu_count_physical_arm_mocked(self):
+ # See: https://github.com/giampaolo/psutil/issues/1359
+ def open_mock(name, *args, **kwargs):
+ if name == '/proc/cpuinfo':
+ return io.BytesIO(textwrap.dedent("""\
+ Processor\t: AArch64 Processor rev 4 (aarch64)
+ processor\t: 4
+ model name\t: AArch64 Processor rev 4 (aarch64)
+ BogoMIPS\t: 26.00
+ BogoMIPS\t: 26.00
+ Features\t: fp asimd evtstrm aes pmull sha1 sha2 crc32
+ CPU implementer\t: 0x41
+ CPU architecture: 8
+ CPU variant\t: 0x0
+ CPU part\t: 0xd03
+ CPU revision\t: 4
+
+ processor\t: 5
+ model name\t: AArch64 Processor rev 4 (aarch64)
+ BogoMIPS\t: 26.00
+ BogoMIPS\t: 26.00
+ Features\t: fp asimd evtstrm aes pmull sha1 sha2 crc32
+ CPU implementer\t: 0x41
+ CPU architecture: 8
+ CPU variant\t: 0x0
+ CPU part\t: 0xd03
+ CPU revision\t: 4
+ """).encode())
+ else:
+ return orig_open(name, *args, **kwargs)
+
+ orig_open = open
+ patch_point = 'builtins.open' if PY3 else '__builtin__.open'
+ with mock.patch(patch_point, create=True, side_effect=open_mock) as m:
+ ret = psutil.cpu_count(logical=False)
+ assert m.called
+ self.assertEqual(ret, 2)
+
@unittest.skipIf(not HAS_CPU_FREQ, "not supported")
def test_cpu_freq_no_result(self):
with mock.patch("psutil._pslinux.glob.glob", return_value=[]):
What's the output of lscpu
?
This is from an 8 processor Linux AArch64/arm64 virtual machine:
Architecture: aarch64 Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 8 Socket(s): 1 NUMA node(s): 1 NUMA node0 CPU(s): 0-7
That is weird. I don't see how it could determine "8" given the content of your /proc/cpuinfo
file. Maybe that means lscpu
relies on something else other than /proc/cpuinfo
? Could you please strace
it, put the output in a txt file and attach it here?
strace lscpu > out.txt
I suspect you also want 2>&1 but it's a few hundred lines of text. Clearly indicates it's reading from lots of things under the /sys/devices/system/cpu/ tree. Do you really want the full strace posted into an issue comment?
@giampaolo the machine @fungi is using is a different arm system than mine. I expect if I ran lscpu
on mine it would return 4 cpus. I just don't have that box handy right now. When I do I'll paste the contents of lscpu from it too.
I suspect you also want 2>&1 but it's a few hundred lines of text.
Correct. Thanks.
Do you really want the full strace posted into an issue comment?
I think github allows you to attach it as a text file.
@giampaolo I just ran lscpu
on my system and got:
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 10
On-line CPU(s) list: 0-3
Off-line CPU(s) list: 4-9
Thread(s) per core: 0
Core(s) per socket: 2
Socket(s): 2
Vendor ID: ARM
Model: 4
Model name: Cortex-A53
Stepping: r0p4
CPU max MHz: 2002.0000
CPU min MHz: 221.0000
BogoMIPS: 26.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32
I also just spun up a raspberry pi 3 b+ running a current aarch64 kernel (v4.19.1) and ran the commands on that too for comparison:
- /proc/cpuinfo contents:
processor : 0
BogoMIPS : 38.40
Features : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
processor : 1
BogoMIPS : 38.40
Features : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
processor : 2
BogoMIPS : 38.40
Features : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
processor : 3
BogoMIPS : 38.40
Features : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4
- python -c "print(repr(open('/proc/cpuinfo').read()))":
'processor\t: 0\nBogoMIPS\t: 38.40\nFeatures\t: fp asimd evtstrm crc32 cpuid\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nprocessor\t: 1\nBogoMIPS\t: 38.40\nFeatures\t: fp asimd evtstrm crc32 cpuid\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nprocessor\t: 2\nBogoMIPS\t: 38.40\nFeatures\t: fp asimd evtstrm crc32 cpuid\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\nprocessor\t: 3\nBogoMIPS\t: 38.40\nFeatures\t: fp asimd evtstrm crc32 cpuid\nCPU implementer\t: 0x41\nCPU architecture: 8\nCPU variant\t: 0x0\nCPU part\t: 0xd03\nCPU revision\t: 4\n\n'
- lscpu:
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
Vendor ID: ARM
Model: 4
Model name: Cortex-A53
Stepping: r0p4
BogoMIPS: 38.40
Flags: fp asimd evtstrm crc32 cpuid
FYI, I cannot reproduce this error on the Amazon Graviton2 processor and a psutil aarch64 wheel built from source:
Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from packaging.markers import default_environment
>>> default_environment()
{'implementation_name': 'cpython', 'implementation_version': '3.8.10', 'os_name': 'posix', 'platform_machine': 'aarch64', 'platform_release': '5.13.0-1017-aws', 'platform_system': 'Linux', 'platform_version': '#19~20.04.1-Ubuntu SMP Mon Mar 7 12:55:31 UTC 2022', 'python_full_version': '3.8.10', 'platform_python_implementation': 'CPython', 'python_version': '3.8', 'sys_platform': 'linux'}
>>> import psutil
>>> (psutil.cpu_count(logical=False), psutil.cpu_count(logical=True))
(4, 4)
CPU info
# cat /proc/cpuinfo
processor : 0
BogoMIPS : 243.75
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x3
CPU part : 0xd0c
CPU revision : 1
processor : 1
BogoMIPS : 243.75
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x3
CPU part : 0xd0c
CPU revision : 1
processor : 2
BogoMIPS : 243.75
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x3
CPU part : 0xd0c
CPU revision : 1
processor : 3
BogoMIPS : 243.75
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x3
CPU part : 0xd0c
CPU revision : 1
# lscpu
Architecture: aarch64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: ARM
Model: 1
Model name: Neoverse-N1
Stepping: r3p1
BogoMIPS: 243.75
L1d cache: 256 KiB
L1i cache: 256 KiB
L2 cache: 4 MiB
L3 cache: 32 MiB
NUMA node0 CPU(s): 0-3
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs