GPTQ quantization version Vicuna?

[OedoSoldier]

Can we use a GPTQ quantized version of Vicuna v0 as the backbone?

[TsuTikgiau]

First, thanks for referring the GPTQ quantized version to us! We don't test this before. We will have a check once we are available for this. Thank you!

[OedoSoldier]

First, thanks for referring the GPTQ quantized version to us! We don't test this before. We will have a check once we are available for this. Thank you!

I've made it run with 4bit GPTQ quantized, and it works fine. Generation is significantly fast but I can observe loss of performance.

[Gavr728]

First, thanks for referring the GPTQ quantized version to us! We don't test this before. We will have a check once we are available for this. Thank you!

I've made it run with 4bit GPTQ quantized, and it works fine. Generation is significantly fast but I can observe loss of performance.

Can you please provide insights on how you have made it?