if LoRAX is based on punica kernels will it be able to support LoRA Adapters for Mistral NeMO 12B?

[tensimixt]

Feature request

if LoRAX is based on punica kernels will it be able to support LoRA Adapters for Mistral NeMO 12B? which has a vocab size > 130k. Currently Vllm for example doesn't support vocab_size > 128512 when enable_lora=True

I think if huggingface and LoRAX are based on punica kernels they will also have this limitation or this limitation does not exist for TGI and LoRAX?

Thank you!

Motivation

be able to run inference with Mistral NeMO + LoRA Adapter (in a multi-lora world)

Your contribution

Checked various deployment providers and found the limitation

[Nero10578]

Feature request

if LoRAX is based on punica kernels will it be able to support LoRA Adapters for Mistral NeMO 12B? which has a vocab size > 130k. Currently Vllm for example doesn't support vocab_size > 128512 when enable_lora=True

I think if huggingface and LoRAX are based on punica kernels they will also have this limitation or this limitation does not exist for TGI and LoRAX?

Thank you!

Motivation

be able to run inference with Mistral NeMO + LoRA Adapter (in a multi-lora world)

Your contribution

Checked various deployment providers and found the limitation

did you figure out if Mistral Nemo 12B works with lora adapters with lorax? It does not work with VLLM or Aphrodite still and I am looking for alternatives.

[preduct0r]

Feature request

if LoRAX is based on punica kernels will it be able to support LoRA Adapters for Mistral NeMO 12B? which has a vocab size > 130k. Currently Vllm for example doesn't support vocab_size > 128512 when enable_lora=True I think if huggingface and LoRAX are based on punica kernels they will also have this limitation or this limitation does not exist for TGI and LoRAX? Thank you!

Motivation

be able to run inference with Mistral NeMO + LoRA Adapter (in a multi-lora world)

Your contribution

Checked various deployment providers and found the limitation

did you figure out if Mistral Nemo 12B works with lora adapters with lorax? It does not work with VLLM or Aphrodite still and I am looking for alternatives.

Did you find alternatives to vllm? I still strugle with this problem of serving mistral_nemo with lora