mistral.rs icon indicating copy to clipboard operation
mistral.rs copied to clipboard

Published 20 hours ago • verified publisher icon EricLBuehler

Is there any plans to support AWQ and GPTQ in the future?

Open franklucky001 opened this issue 1 year ago • 2 comments

support quantization of awq and gptq

franklucky001 avatar Jun 11 '24 06:06 franklucky001

Hi @franklucky001! I do plan on supporting other quantization method such as AWQ and GPTQ in the future. I would probably start with GPTQ.

EricLBuehler avatar Jun 13 '24 05:06 EricLBuehler

@franklucky001 we are going to merge GPTQ soon!

EricLBuehler avatar Jul 04 '24 09:07 EricLBuehler

@franklucky001 GPTQ and Marlin support have been merged already.

EricLBuehler avatar Nov 28 '24 20:11 EricLBuehler

So does AWQ work? For some reason I could've sworn it was a form of GPTQ so I figure there's a possibility. Wish I had an AWQ model downloaded to try

BuildBackBuehler avatar Dec 07 '24 01:12 BuildBackBuehler