GPU Usage dropping before completion ends

[jeanromainroy]

I have been using the new Command-R+ model in 4-bit mode and consistently observe a drop in GPU utilization immediately after prompt evaluation, as it begins generation/prediction. This leads to significantly reduced performance.

During evaluation:

During generation – drop occurs right before the first token is predicted (i.e. "<PAD>"):

Here's my setup: Machine: Apple M2 Ultra (cores: 8E+16P+60GPU), 192GB Ram ProductName: macOS ProductVersion: 14.3 BuildVersion: 23D56

I have tried with and without setting my memory limit: sudo sysctl iogpu.wired_lwm_mb=150000

I have tried with and without disabling the cache: mx.metal.set_cache_limit(0)

Any help would be welcome, because at the moment I am only able to use the llama.cpp implementation of Command-R+, which works without any issues.