worker-vllm icon indicating copy to clipboard operation
worker-vllm copied to clipboard

add multi lora support

Open sven-knoblauch opened this issue 1 year ago • 4 comments

in addition to the last pull reqeust, now you can also use multiple lora adapters.

Now you can use a single adapter as input (as before): {"name": "xxx", "path": "xxx/xxxxx", "base_model_name": "xxx/xxxx"}

And also as a list of adapters: [{"name": "xxx", "path": "xxx/xxxxx", "base_model_name": "xxx/xxxx"},{"name": "xxx", "path": "xxx/xxxxx", "base_model_name": "xxx/xxxx"},...]

sven-knoblauch avatar Nov 04 '24 09:11 sven-knoblauch

Thanks @pandyamarut for merging the last lora adapter Pull Request. Just added a small update to include also multi lora support.

sven-knoblauch avatar Nov 04 '24 09:11 sven-knoblauch

Hey mind writng some simple examples to the readme.md too? would be nice if theres an example usage

nerdylive123 avatar Nov 07 '24 16:11 nerdylive123

Waiting for merge please. Seems the LoRA support is broken due to some conflicts since v1.9 https://github.com/runpod-workers/worker-vllm/blob/6fc770415def3b65ddf8c3a80a8b36fb1454f8e7/src/engine.py#L145-L151

BeautyyuYanli avatar Feb 11 '25 09:02 BeautyyuYanli

should be working now with the new version

sven-knoblauch avatar Feb 18 '25 08:02 sven-knoblauch