Inquiry About Support for Evaluating AWQ Quantized Models

[donghong1]

Hi, Thank you for your great work on this repository! I am currently exploring the use of AWQ for model quantization and was wondering if the repository supports evaluating models that have been quantized using this technique.

[sasha-hailo]

The repo definitely supports running AWQ quantization and evaluating the quantized models. Take a look on the example configuration files for running AWQ in folder https://github.com/ModelTC/llmc/tree/main/configs/quantization/methods/Awq

[Disclaimer: I'm not an author of this repo]