casper-hansen
GPU not detected
having issues with autoAWQ not detecting my RTX 3070ti with Cuda 12.4 installed when attempting to run with loading "TheBloke/Mistral-7B-OpenOrca-AWQ"
the command nvidia-smi returns:
Sun Apr 7 09:39:24 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 551.86 Driver Version: 551.86 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3070 Ti WDDM | 00000000:01:00.0 On | N/A |
| 0% 43C P8 18W / 290W | 571MiB / 8192MiB | 4% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
The 3 specific error messages are:
ValueError: Could not load model TheBloke/Mistral-7B-OpenOrca-AWQ with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 'transformers.models.mistral.modeling_mistral.MistralForCausalLM'>). See the original errors:
ValueError: AWQ is only available on GPU
ValueError: AWQ is only available on GPU
have got past the above by reinstalling torch which for some reason did not initially install cuda support. but now it is telling me "You have loaded an AWQ model on CPU and have a CUDA device available, make sure to set your model on a GPU device in order to run your model."