Please add lora support for higher ranks and alpha values

[parikshitsaikia1619]

ValueError: LoRA rank 64 is greater than max_lora_rank 16.

[SuperBruceJia]

Mark

[Peter-Devine]

Bump

[dspoka]

It's not super well documented but you need to just pass in "-max-lora-rank 64" or whatever when serving since default is 16.

python -m vllm.entrypoints.openai.api_server --max-lora-rank 64
--model model_name
--enable-lora
--lora-modules lora-name=lora_path

[spreadingmind]

It's not super well documented but you need to just pass in "-max-lora-rank 64" or whatever when serving since default is 16.

python -m vllm.entrypoints.openai.api_server --max-lora-rank 64 --model model_name --enable-lora --lora-modules lora-name=lora_path

Thanks for the answer, it helped me as well. For those who use code, it would be here:

          llm = LLM(
            model=args.model, tensor_parallel_size=torch.cuda.device_count(), 
            dtype=args.dtype, trust_remote_code=True, enable_lora=True, max_lora_rank=64
        )

[Napuh]

Both answers work for me, up to rank 64. Rank > 64 is not supported yet.

See #3934

[patrickrho]

Can we get Lora rank > 64 supported and merged?

edit: im also curious if this was by design to support up to 64 rank, if so please let me know

[kevinjesse]

Bump. I need adapters that are much, much larger to be supported. Thanks

[jiangjin1999]

Is there something special about lora rank >64. Wonder why only <=64 is supported

[JohnUiterwyk]

same here, this is a blocker for me

[Peter-Devine]

@JohnUiterwyk , has this not been fixed by the suggestions from @dspoka and @spreadingmind ? Their suggestions worked for me.

[JohnUiterwyk]

No, as the maximum max_lora_rank is 64, going higher than that throws an error. I have adapters with rank 128 and 256 for certain uses cases, and can not serve them with vllm as a result of the hardcoded limit for the allowed value passed to max_lora_rank

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[AntreasAntonio]

Any updates on this? Recent papers showed that a rank=256 seems to be very beneficial for example. I suspect this trend will continue to be the case and increasing in the near future.

[Jiaxin-Wen]

Any updates on this? Recent papers showed that a rank=256 seems to be very beneficial for example. I suspect this trend will continue to be the case and increasing in the near future.

interesting, do you know the paper link @AntreasAntonio

[Vaibhav-Sahai]

wanted to update: LORA rank 256 works with vllm 0.8.2

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions[bot]]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!