Quantization support

[generalsvr]

How to use 8bit quantized models? Can I run GGML/GGUF models?

[hiworldwzj]

8bit weightonly quantized only support llama now

[generalsvr]

Any examples?

[hiworldwzj]

parser.add_argument("--mode", type=str, default=[], nargs='+',
                    help="Model mode: [int8kv] [int8weight | int4weight]")

[XHPlus]

As for the model file format, we have not tested GGML/GGUF up to now. What is the motivation to use these formats?

[JustinLin610]

Will GPTQ be supported?

[suhjohn]

@XHPlus There's a lot of open source models on HuggingFace driven by https://huggingface.co/TheBloke. Many people in the open source community use those quantized models on TGI / vLLM.

[adi]

parser.add_argument("--mode", type=str, default=[], nargs='+',
                    help="Model mode: [int8kv] [int8weight | int4weight]")

Using this option with Llama2-13B gives this error:

_get_exception_class.<locals>.Derived: 'LlamaTransformerLayerWeightQuantized' object has no attribute 'quantize_weight'

I tried both --mode int8kv int4weight and --mode int8kv int4weight

Any suggestions how to fix this?

[VfBfoerst]

@XHPlus Quantization is partially the only way to run bigger models in smaller GPUs, e.g. Mixtral. With vLLM, I can run mixtral quantized with 48 GBs of VRAM. The unquantized model would use up to 100GB VRam i guess.