about NVidia GPU use example

[CrazyJson]

I have an RTX 4060 graphics card, how do I deploy a gpu version of the model with this project

[CrazyJson]

[martindevans]

You need a gguf model file to use llama.cpp, not safetensors.

[CrazyJson]

Thanks, I understand llama.cpp is used to load the quantized gguf model. One more question, which parameter in the sample code is used to enable the use of the local GPU, and how to choose which local gpu to use

[ChengYen-Tang]

@CrazyJson , You need install coda in your pc if you install cuda11, you should be choose this package cuda12 use this package

I'm not sure if OpenCL supports Intel graphics cards

[ZCOREP]

i have the same problem, i download and install Cuda12 , but not use my GPU still use RAM!

[martindevans]

do you have the cuda toolkit installed? You need that to supply the CUDA runtime packages

[ZCOREP]

yes i did