Error processing prompt (see logs with DEBUG>=2): module load failed with status code 222: CUDA_ERROR_UNSUPPORTED_PTX_VERSION

[Pentar0o]

I have this error Error processing prompt (see logs with DEBUG>=2): module load failed with status code 222: CUDA_ERROR_UNSUPPORTED_PTX_VERSION : trying Llama 3.1 70b on a server with Cuda 12.6 / Nvidia driver 535.183.06, with 3 Tesla T4 Gpu, any help appreciated.

[pickettd]

I had this issue but when I scrolled up in the logs it told me that ptxas executable was not found. I had to add my cuda bin to my shell path. Also I had to make sure that my nvidia driver version and cuda toolkit versions matched (as shown by running nvidia-smi and nvcc --version). Once I updated my path, updated my drivers, and restarted the machine, I got past the error