AutoAWQ icon indicating copy to clipboard operation
AutoAWQ copied to clipboard

Does AutoAWQ support multi-threading CPU?

Open sdecoder opened this issue 1 year ago • 0 comments

Greetings everyone.

  1. Server configuration: a modern CPU with multiple cores/large memory/a relatively weak GPU with insufficient VRAM(16G);
  2. In such case, it is impossible to use the GPU to quantize the model due to low VRAM;
  3. So is it possible to leverage the multiple cores of CPU?
  4. I have tried to set the device_map="cpu", the work has been bound to CPU, but with ONLY ONE core. So it runs really slowly.

Can anyone provide any idea/hint to solve this issue? Thank you all :)

sdecoder avatar Aug 23 '24 16:08 sdecoder