[MacOS] psutil.cpu_freq() broken on Apple M1

[Lomanic]

Summary

• OS: MacOS
• Architecture: Apple M1
• Psutil version: N/A
• Python version: N/A
• Type: core

Description

Sorry that I couldn't provide some of the info in the template as I don't own an Apple M1 device, but it is highly suspected that calling psutil.cpu_freq() on Apple M1 is failing as the hw.cpufrequency sysctl call (used here) was removed on this arch (can be checked with sysctl hw.cpufrequency on any Apple M1 device, while it is working on amd64).

See https://github.com/shirou/gopsutil/pull/999 and https://github.com/shirou/gopsutil/issues/1000

In https://github.com/shirou/gopsutil/pull/999 a fix was suggested but I couldn't confirm it.

psutil user-base is probably bigger than gopsutil's so I hope this will help both projects find a solution or confirm @shoenig fix in his PR.

[giampaolo]

Mmm... is "hw.cpufrequency" the only one that fails? Later on we also use "hw.cpufrequency_min" and "hw.cpufrequency_max". Do they fail as well?

Sorry that I couldn't provide some of the info in the template

It's OK, no worries.

[tmm1]

None are present.

tmm1@Mac-mini ~ % sysctl hw.cpufrequency
tmm1@Mac-mini ~ % sysctl hw.cpufrequency_min
tmm1@Mac-mini ~ % sysctl hw.cpufrequency_max
tmm1@Mac-mini ~ %

[giampaolo]

For a possible solution see https://github.com/giampaolo/psutil/issues/975 pointing to https://stackoverflow.com/a/14355710/376587 (use sysctl + HW_CPU_FREQ or sysctl + KERN_CLOCKRATE). It looks like both approaches report the advertised frequency (it never changes), same as the current code.

[tmm1]

tmm1@Mac-mini /tmp % cat freq.c 

#include <stdio.h>
#include <sys/sysctl.h>

int main() {
        int mib[2];
        unsigned int freq;
        size_t len;

        mib[0] = CTL_HW;
        mib[1] = HW_CPU_FREQ;
        len = sizeof(freq);
        sysctl(mib, 2, &freq, &len, NULL, 0);

        printf("%u\n", freq);

      mib[0] = CTL_KERN;
      mib[1] = KERN_CLOCKRATE;
      struct clockinfo clockinfo;
      len = sizeof(clockinfo);
      sysctl(mib, 2, &clockinfo, &len, NULL, 0);
      printf("clockinfo.hz: %d\n", clockinfo.hz);
      printf("clockinfo.tick: %d\n", clockinfo.tick);

        return 0;
}
tmm1@Mac-mini /tmp % gcc -o freq freq.c 
tmm1@Mac-mini /tmp % ./freq
0
clockinfo.hz: 100
clockinfo.tick: 10000

tmm1@Mac-mini /tmp % sysctl kern.clockrate
kern.clockrate: { hz = 100, tick = 10000, tickadj = 0, profhz = 100, stathz = 100 }

[giampaolo]

According to doc we should return 0 if min and max frequency cannot be determined, so we can ignore those on M1. This was tracked in #1456 (we return error instead of 0) which should also be fixed.

[giampaolo]

Fixed by https://github.com/giampaolo/psutil/pull/1895.

[Lomanic]

psutil still uses hw.cpufrequency from what I see (but using the mib mechanism), this is still returning 0 on Apple M1 as per https://github.com/giampaolo/psutil/issues/1892#issuecomment-748491277, so I'm quite confused by the fix. Is the fix that now it returns 0 instead of returning an error?

[giampaolo]

Oh you're right. I misread https://github.com/giampaolo/psutil/issues/1892#issuecomment-748491277.

[giampaolo]

The problem is that on my virtualized macOS I get this:

HW_CPU_FREQ: 2590000000 KERN_CLOCKRATE -> clockinfo.hz: 100

...so it appears they are 2 different things.

[ccrvlh]

Same happening in Ubuntu 20.04

[dehydratedpotato]

This is my first time posting in the psutil repo, but I have been trying to figure out how to pull the current CPU frequency on Apple Silicon for a few months now, and I've finally managed to create a script that can do that.

In case anyone wants to take a look at it, the entire project is here, but this is the important part.

Good day 👍

[piskvorky]

I see this wasn't really fixed in #1895 – I'm still getting:

>>> psutil.__version__
5.9.0
>>> psutil.cpu_freq()
FileNotFoundError: [Errno 2] No such file or directory (originated from sysctl(HW_CPU_FREQ))

on M1 (MBP 2021, OS 12.2.1). Let me know how I could help with testing!

[dbwiddis]

I just took a grand tour of almost every GitHub issue regarding this, and think I've cracked this nut.

This comment pointing to the clockrate.hz field is relevant, but the problem is what to multiply it by. The answer appears to be the time base frequency.

On M1s, sysctl hw.tbfrequency returns 24,000,000 which when multiplied by 100 Hz gives the expected 2.4 GHz. However, that sysctl is not reliable on Intel hardware: on my Intel chip I get 1,000,000,000.

This would obviously need testing by anyone with an M1 (or M2?) but I think the general logic should be:

• check if sysctl hw.cpufrequency returns. If so, use it.
• If not, multiply sysctl hw.tbfrequency by sysctl kern.clockrate (hz).

[shoenig]

@dbwiddis FYI I went down this rabbit hole in https://github.com/shirou/gopsutil/pull/999/files

[dbwiddis]

@dbwiddis FYI I went down this rabbit hole in https://github.com/shirou/gopsutil/pull/999/files

Indeed, and yours was one of the ones I visited in my grand tour, but I wasn't sure why you gave up on it; in the example given (a Hackintosh VM) the getFrequencyIntel() call (using hw.cpufrequency) succeeded, so getFrequencyArm() wouldn't. And there appears to be a comment here that implied the test using the HW_CPU_FREQ sysctl was the same thing, but it wasn't. The HW_CPU_FREQ MIB approach is essentially the same ashw.cpufrequency.

However, that said, I think I'm back to agreeing with you that it can't be done.

There are ample ioreg -l outputs on ARM hardware out there for various platforms (iPhones, iPads) and the timebase-frequency appears consistent. See this gist for an iPhone 4 with a known 800 MHz frequency. (Note the byte arrays values shown are little-endian.)

    | +-o cpu0@0  <class IOPlatformDevice, registered, matched, active, busy 0, retain 12>
    | |   {
    | |     "timebase-frequency" = <00366e01>
    | |     "clock-frequency" = <0008af2f>
    | |     "compatible" = <"ARM,cortex-a9","ARM,v7">
    | |   }

Time base frequency 0x016e3600 = 24,000,000 which disproves my proposal.... it's just the useless default.

However, clock-frequency DOES work: Clock frequency 0x2faf0800 = 800,000,000 = the 800 MHz frequency of the Cortex a9.

But clock-frequency isn't reliable. Those same numbers on an M1:

    | +-o cpu0@0  <class IOPlatformDevice, id 0x10000010f, registered, matched, active, busy 0 (175 ms), retain 8>
    | | | {
    | | |   "clock-frequency" = <00366e01>
    | | |   "timebase-frequency" = <00366e01>
    | | |   "compatible" = <"apple,icestorm","ARM,v8">
    | | | }

So, yeah. Hardcoding 2.4 GHz seems to be the thing to do.

[dehydratedpotato]

@dbwiddis 2.4GHz is the timebase frequency, which is not the same as CPU frequency. Apple Silicon has different speeds per cluster. I.e an M1 truly has a PCPU Frequency of 3204 MHz, and a ECPU frequency of 2064 MHz.

M2 chips can have a single core in the PCPU boost to ~3500MHz and the entire ECPU is upped to 2424Mhz.

So if timebase freq is the true goal, then 2.4Ghz may be logical to hardcode.

Otherwise, if you want the real CPU freq, you'll need to pull data from the voltage-states properties in the IORegistry PMGR. Specifically voltage-states1-sram and ``voltage-states5-sram`. (The data there is 32bit little endian and should result in an array. The biggest value is the CPU freq.)

Hopefully that's helpful 👍

[dbwiddis]

Hey @BitesPotatoBacks ... thanks for the cool info. First thread I've seen that on.

I don't actually have an M1 to test on so I'm having to rely on output from others. I did find this gist which is an M1 Macbook Air. I see voltage-states1-sram with 30 32-byte values in it, the largest being 0029f9be which appears to work out to 3.204, so this is useful for getting the max frequency, and it appears there are 15 frequency values (states) that it can be set to.

What I'm really looking for the "nominal" frequency. The output of powermetrics, for example on my (Intel) MBP tells me one of my CPUs is:

CPU Average frequency as fraction of nominal: 119.72% (2753.45 Mhz)

I know my nominal frequency is 2.3 GHz (which sysctl hw.cpufrequency also tells me) and this output indicates that as well by reversing the calculation (2.75345/1.1972 = 2.2999).

Which of the 15 voltage states on an M1 is 'nominal'?

[dehydratedpotato]

@dbwiddis the highest voltage state value (0029f9be, 3204 for PCPU) is actually the nominal in this case, as Apple Silicon does not exhibit a turbo boost behavior like that of Intel.

[dbwiddis]

I guess I'll do that then. In the sample output I only saw a voltage-states5-sram value, ~not one for voltage-states1-sram~ (found it). Is that normal? Are these documented anywhere?

[dehydratedpotato]

@dbwiddis it would be normal to see both ...5-sram and ...1-sram. All apple silicon systems should have those, as command line tool powermetrics relies on those properties for M1 systems.

From research, these voltage-states properties seem to be undocumented, but based on testing, I have determined that

• voltage-states1-sram is the freqs for the ECPU
• voltage-states5-sram is the freqs for the PCPU

On systems with multiple E and P clusters, you may see new voltage-states. For example, on an M1 Pro/Max, you may see voltage-states13-sram. It is also for the PCPU but is not necessary to pull info from, as both P-clusters use the same freqs...

[dbwiddis]

Got it, and thanks!

[dehydratedpotato]

@dbwiddis no problem! 👍

[johnny12150]

I see this wasn't really fixed in #1895 – I'm still getting:

>>> psutil.__version__
5.9.0
>>> psutil.cpu_freq()
FileNotFoundError: [Errno 2] No such file or directory (originated from sysctl(HW_CPU_FREQ))

on M1 (MBP 2021, OS 12.2.1). Let me know how I could help with testing!

Still facing this issue with M2 Max and psutil 5.9.4

>>> import psutil
>>> psutil.cpu_freq()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/john/miniconda3/envs/pytorch-m2/lib/python3.8/site-packages/psutil/__init__.py", line 1864, in cpu_freq
    ret = _psplatform.cpu_freq()
  File "/Users/john/miniconda3/envs/pytorch-m2/lib/python3.8/site-packages/psutil/_psosx.py", line 179, in cpu_freq
    curr, min_, max_ = cext.cpu_freq()
FileNotFoundError: [Errno 2] No such file or directory (originated from sysctl(HW_CPU_FREQ))

[k-nearest-neighbor]

Same on 5.9.4 on an M1 Pro with Ventura 13.2.1

>>> psutil.__version__
'5.9.4'
>>> psutil.cpu_freq()
FileNotFoundError: [Errno 2] No such file or directory (originated from sysctl(HW_CPU_FREQ))

[snOm3ad]

@giampaolo I have seen this issue fixed in other OSS projects. For example, osquery retrieves the voltage-states5-sram from the ARM IO device in the IO Kit registry then walks through the array, 4-bytes at a time and returns the maximum as the CPU frequency:

https://github.com/osquery/osquery/blob/042cb999f298e81da610bd630ae24bcf0740296c/osquery/tables/system/darwin/cpu_info.cpp#L124-L128

On my machine, this leads a max frequency of 3.5Ghz which matches what I see in powermetrics. I can prepare a PR with these changes, I just need some guidance in terms of where to put the IO Kit traversal code?

[giampaolo]

Sure, please go for it!

I just need some guidance in terms of where to put the IO Kit traversal code?

I would say psutil/arch/osx/cpu.c. We already use IOKit elsewhere but we didn't put it into a specific/separate file:

$ grep.py Kit
psutil/_psutil_osx.c                                                                                                                                                                          
 31   │ #include <IOKit/IOKitLib.h>
 32   │ #include <IOKit/storage/IOBlockStorageDriver.h>
 33   │ #include <IOKit/storage/IOMedia.h>
 34   │ #include <IOKit/IOBSD.h>
 35   │ #include <IOKit/ps/IOPowerSources.h>
 36   │ #include <IOKit/ps/IOPSKeys.h>

[shoenig]

For reference here's a simple C version we use for Go https://github.com/shoenig/go-m1cpu/blob/main/cpu.go

[dbwiddis]

And here's a Java/JNA based solution based on this Issue/thread: https://github.com/oshi/oshi/blob/e78a0ffb32d5bc2fe6943b903ed5eeb97f949fc5/oshi-core/src/main/java/oshi/hardware/platform/mac/MacCentralProcessor.java#L322-L357

[snOm3ad]

PR is up, could someone please review it? Thank you @shoenig and @dbwiddis both your examples where useful.

[SansGuidon]

I'm also interested by thix fix, as on M1, without that, no way it seems to use BabyAGI :-) (https://github.com/oliveirabruno01/babyagi-asi)

 ✘  py[babyagi-asi]  ~/c/babyagi-asi   main   python babyagi.py
Traceback (most recent call last):
  File "/Users/xxx/Code/babyagi-asi/babyagi.py", line 1, in <module>
    import openai, prompts, consts, pinecone, os, subprocess, tiktoken, json
  File "/Users/xxx/Code/babyagi-asi/prompts.py", line 9, in <module>
    I am running on a {platform.system()} {platform.architecture()[0]} system with {round(psutil.virtual_memory().total / (1024 ** 3), 2)} GB RAM and a {psutil.cpu_freq().current/1000 if psutil.cpu_freq() else "unknown"} GHz CPU. I am using OpenAI API. I must remember to use '|' instead of '&&' or '&' in my commands if using windows' cmd or pws.
  File "/Users/xxx/Library/Python/3.9/lib/python/site-packages/psutil/__init__.py", line 1864, in cpu_freq
    ret = _psplatform.cpu_freq()
  File "/Users/xxx/Library/Python/3.9/lib/python/site-packages/psutil/_psosx.py", line 179, in cpu_freq
    curr, min_, max_ = cext.cpu_freq()
FileNotFoundError: [Errno 2] No such file or directory (originated from sysctl(HW_CPU_FREQ))

However this works: in the requirements.txt file, replace the line with that dependency, e.g

psutil==5.9.4

with

git+https://github.com/snOm3ad/psutil.git@fix-cpu-freq-apple-silicon

then pip install ...