Casper

This is a pytorch-related issue. You are installing wheels that are built with 2.2.0 and trying with torch 2.3.0. To fix this, you need to build from source or wait...

@BenjaminBossan yes. I am building new kernels now and pushing an updated version to PyPi. Sorry for the delay - I plan to be faster at updating when new torch...

Hi @kwonjihun-theori, thanks for your very detailed post. I met the same challenges as you when trying to apply activation-aware quantization to this model as it's a non-standard model definition....

Hi @Zephyr69, I am happy to investigate this issue. Do you have an example of how to trigger the jibberish? Otherwise, I would advise you to use [vLLM](https://github.com/vllm-project/vllm) to serve...

You will probably have to quantize your model using a custom dataset for coding.

I see this is a draft PR. @TechxGenus have you done any further testing?

We don't support lora modules. You would have to convert your model to standard weights

Sorry for taking so long. I took leave from working on open-source the past month.

I plan to support new models as they come out. I focus most of my efforts on models that are better than previous models and mostly let others add older...

Hi @user-ZJ, I would love to support more models. There is no documentation yet, but that is subject to change as AutoAWQ now has a dedicated documentation page. Do you...