Windows: GPU is not taxed enough compared to other apps/frameworks

[ramz35]

I have a cuda compiled llama-cpp-python and on start up I can see gpu correctly identified and all 33 layers offloaded onto GPU. I can see the model fits into my VRAM. If I run the default model or one of my local ones (eg. TheBloke-LLama2-Chat-7B) the GPU doesn't go past say 38% while the CPU is max 20%. In other apps/frameworks eg. h2ogpt, llama.cpp the GPU is maxed out while the CPU is similar or lower.

Just curious if this is a bug or a non-optimal version of the code or if working as designed?

edit: there was another issue (closed) that mentioned an untaxed GPU but with CPU at full pelt but that apparently was fixed with version 0.3.0 which I have, and it's not exactly the same as mine as my CPU isn't taxed.

[Zirgite]

I have similar results with Mistral 7B.

[paul-asvb]

+1 on ubuntu