mlx-examples
mlx-examples copied to clipboard
GPU Usage dropping before completion ends
I have been using the new Command-R+ model in 4-bit mode and consistently observe a drop in GPU utilization immediately after prompt evaluation, as it begins generation/prediction. This leads to significantly reduced performance.
During evaluation:
During generation – drop occurs right before the first token is predicted (i.e. "<PAD>"):
Here's my setup: Machine: Apple M2 Ultra (cores: 8E+16P+60GPU), 192GB Ram ProductName: macOS ProductVersion: 14.3 BuildVersion: 23D56
I have tried with and without setting my memory limit:
sudo sysctl iogpu.wired_lwm_mb=150000
I have tried with and without disabling the cache:
mx.metal.set_cache_limit(0)
Any help would be welcome, because at the moment I am only able to use the llama.cpp implementation of Command-R+, which works without any issues.