Jee Jee Li
Jee Jee Li
> Also cc @HwwwwwwwH . Are there any existing LoRA weights for this model that can be used to test this feature? We have a local version that only adds...
Changing the status to trigger the lora test. After completing the lora test, I will convert the status to WIP
> > Also cc @HwwwwwwwH . Are there any existing LoRA weights for this model that can be used to test this feature? > > No such thing, as I...
@DarkLight1337 Unlike LLM, the LoRA unit test for VL models has inputs that include not only text but also image. Where should I place the image?
@DarkLight1337 After several days of investigation, I found that the current method of integrating LoRA in vLLM does not support adding LoRA to both the visual encoder and llm simultaneously....
@DarkLight1337 Sorry for the delayed feedback - In the current LoRA implementation, all LoRA layers within a single model share the same LoRA [metadata](https://github.com/vllm-project/vllm/blob/v0.5.4/vllm/lora/models.py#L462). However, the vision module and the...
Without changing the existing LoRA pipeline, the following modifications may be necessary: - Differentiate whether the current LoRA layer belongs to the vision module or the LLM module, and then...
Currently, I have implemented an interim solution to enable LoRA support for multimodal models. This solution only supports LoRA inference for the language model component. For LoRA in visual modules(or...