feat: Add MacOS metal monitoring on Mac

[hiro-v]

Problem I want to see my MacOS metal gpu monitoring (GPU utilization/ VRAM) on Jan to check the resource consumption of models use

Success Criteria

• Correctly show the utilization
• Can see the UI at bottom bar

Additional context

• https://github.com/automation-stack/electron-sudo
• Use powermetrics and parse result

sudo powermetrics -i 1000 --samplers all --show-usage-summary

[Foul-Tarnished]

https://github.com/op06072/NeoAsitop

This would probably be better than powermetrics, because NeoAsitop doesn't require admin privileges (sudo)

[hiro-v]

Thank you @Foul-Tarnished . Awesome! I have traced the code back to the snippet: https://github.com/dehydratedpotato/socpowerbud/tree/main/socpwrbud I think building a small library in cpp and bind it with node-gyp and make it available in nodejs might be a good way to integrate with the current Jan app.

[0xgokuz]

Can I take this?

[0xgokuz]

Hi @imtuyethan, I think we'll need the UI/UX design from you for this issue as well. The big UI/UX question that we have is that whether we should be requesting the User for their sudo password every time?

The options for the monitoring tools:

1. https://github.com/dehydratedpotato/socpowerbud
2. https://github.com/op06072/NeoAsitop
3. https://github.com/tlkh/asitop
4. https://github.com/automation-stack/electron-sudo

Given that we want a good UI/UX, we don't want the user to keep sudo-ing everytime. So it's likely we would just go with what @Foul-Tarnished suggested https://github.com/op06072/NeoAsitop

[Foul-Tarnished]

Can I take this?

Another important metric would be this : By default, MacOS let its GPU only use ~65% of the RAM

in Faraday app, they show this above, useful to know the biggest models that can run on your GPU with Metal backend.

Thankfully, we can increase this with sudo sysctl iogpu.wired_limit_mb=30720 (set to 0 for default behavior) sadly it needs admin privileges, but maybe we could at least read and display the value without sudo ? not sure it works

(in faraday they just do 0.65* ram, they don't read the actual value, so if you tweak it manually, it won't try to run bigger models with metal, even though it could, so you also need to force enable metal backend in the app)

With this tweak, I can run Mixtral8x7b (25GB model) on the gpu, really fast, on a 36GB macbook model (even with +2k context)

[Foul-Tarnished]

Another cool metric would be watts usage, especially if you are running on the battery

[freelerobot]

Outdated issue