AutoAWQ
AutoAWQ copied to clipboard
Auto detect if GPU is unsupported
The error:
RuntimeError: CUDA error: no kernel image is available for execution on the device
Implies but does not immediately explain that it's due to old architecture. I propose either displaying a warning such as "The GPU might fail" if Compute Capability is below 7.5 or running any torch command that will fail and if it fails displaying "The GPU will fail" sort of message.
Ran into the same problem and figured out it's because the GPU card isn't supported. For people who wish to try AWQ quantized models, you can check your GPU's compute capability on this CUDA website: NVIDIA CUDA GPUs. Might save some time for anyone else bumping into this issue.