llama-cpp-python icon indicating copy to clipboard operation
llama-cpp-python copied to clipboard

Published 20 hours ago • verified publisher icon abetlen

llama-server not using GPU

Open RakshitAralimatti opened this issue 1 year ago • 5 comments

After I install llama-cpp-python-server with cuda support and run python3 -m llama_cpp.server --model starcoderbase-3b/starcoderbase-3b.Q4_K_M.gguf --n_gpu_layers 10 The GPU is not getting used its running on the CPU

RakshitAralimatti avatar Nov 09 '24 04:11 RakshitAralimatti