model not loading on GPU

[kot197]

Hi, I just want to run this simple code on GPU...

from langchain_community.llms import CTransformers

llm = CTransformers(model="./airoboros-mistral2.2-7b.Q4_K_S.gguf", model_type="mistral", gpu_layers=32, verbose=True)

print(llm.invoke('AI is going to'))

As you can see...GPU usage is at 0% and it took about 1 minute which is very long for a quick prompt

Is there nothing I can do to visualize what's going on, like printing something on the terminal? How do I even know if it's running or not? I cannot fully trust task manager

I'm a newcomer, please help me