llm-course icon indicating copy to clipboard operation
llm-course copied to clipboard
Published 6 months ago • verified publisher icon mlabonne

i-Quants in AutoQuant?

Open GameOverFlowChart opened this issue 1 year ago • 2 comments

Would it be possible to support i-Quants in AutoQuant or are they more demanding to quantize?

GameOverFlowChart avatar Apr 27 '24 23:04 GameOverFlowChart

Can't you already create iquants by providing the right name? https://github.com/ggerganov/llama.cpp/blob/04976db7a819fcf8bfefbfc09a3344210b79dd27/gguf-py/gguf/constants.py#L811

mlabonne avatar May 07 '24 16:05 mlabonne

Oh right in this case this should be added to the list of the names that is shown in the notebook, at least one of them as an example (so that you see at first glance that there is no underscore between I and Q.

GameOverFlowChart avatar May 10 '24 12:05 GameOverFlowChart