janhq
Discussion: How to tell if Nitro is using a GPU?
Motivation
I'm tinkering with Nitro for the first time. I've successfully loaded a small model, however responses from the completions route are very slow (I'm minutes in and have yet to receive a response).
Discussion
I don't know if Nitro is actually using the GPU; it's definitely using all of my CPU so I'm guessing not. Is there some string that I can look for in stdout or stderr? For what it's worth, I've verified that other, non-Nitro run in the same fashion does indeed use the GPU.
Resources
I'm using the example commands here: https://github.com/janhq/nitro/blob/d0cc50b893609d66006cf8d2b24f0568cd687ad5/README.md
if you are using "ngl" with value ranging from 0-3x and using a distribution of binary with cuda , it's using gpu, you can tell from the log also when model is loaded up
There seems to be a problem with the install script since version 0.2.5 where the incorrect download URL is used. There are now 2 different cuda releases:
But the download URL leaves off the -11-7 or -12-0 and only appends -cuda. We were able to navigate around this but updating the install.sh manually to include the -12-0 postfix. Either the script needs to be updated to allow for more granular selection or one of the created cuda releases needs to be assigned as a default for the script to work.
i'd recommend not use install script since it's not recently updated, using pure binary file will give you the best experience