Ryan Shrott

@pngwn thank you. Adding to the "Use via API" docs is perfect. Do you know the timeline on that? Will it be in the next release?

@borzunov Is that local or HF model hub? I'm looking for HF model hub option.

Will this be added to main VLLM package soon? I get this error now: llm = LLM(model="rshrott/description-awq-4bit") RuntimeError: Cannot find any model weights with `rshrott/description-awq-4bit`

Thanks. How do I compile the PR? Is this not currently available for general usage?

I'm trying to do in google colab: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) [](https://localhost:8080/#) in () ----> 1 llm = LLM(model="rshrott/description-awq-4bit", **{'quantization': 'awq'}) [/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py](https://localhost:8080/#) in __init__(self, model, tokenizer, tokenizer_mode,...

Yes please