vllm
vllm copied to clipboard
vllm-project
How can I use the Lora Adapter for a model with Vocab size 40960?
The error occurs when I call the LLMEngine object. The error below appears.
ValueError: When using LoRA, vocab size must be 32000 >= vocab_size <= 33024
Does the method using the Lora Adapter not apply to models that expand the vocab or have a vocab size larger than 32000?
same issue, I deployed a LLaMA structure LLLM but the vocab_size had been expanded to 130,000
same problem, please add support for vocab_size over 32000. in my case need vocab_size 60256
I found a solution which inspired by Gemma you can find the file: vllm/vllm/model_executor/models/llama.py, then find the definition of LlamaForCausalLM (in line 278)
that's the original support lora modules
# LoRA specific attributes
supported_lora_modules = [
"qkv_proj",
"o_proj",
"gate_up_proj",
"down_proj",
"embed_tokens",
"lm_head",
]
embedding_modules = {
"embed_tokens": "input_embeddings",
"lm_head": "output_embeddings",
}
embedding_padding_modules = ["lm_head"]
just remove some modules which may cause errors and finally saving like this:
# LoRA specific attributes
supported_lora_modules = [
"qkv_proj",
"o_proj",
"gate_up_proj",
"down_proj",
]
embedding_modules = {}
embedding_padding_modules = []
Why doesn't the lora adapter load based on the vocabulary size of the origin model, even though the target_modules in adapter_config.json doesn't have embed_tokens and lm_head?
Was also trying to run Yi with the same problem.. @Yard1 can you elaborate on what's needed to support? I'd be glad to work on this PR
We'd need to modify the kernel/write a new LoRA kernel to support such a large vocabulary size. The current Punica ones cannot do that.
We just don't allow the embeddings layer to be modified on gemma
The Yi_6B has 64000 vocab_size, when i modify the llama model source just like gemma，it's work fine。
but the Yi_6B_200K always throw exception:
The Yi_6B_200K has a different max_position_embeddings:200000
@Yard1
+1. Need support for a larger vocabulary size. In addition to vocabulary size, there is also a use case like LLaMA Pro block expansion fine-tuning with LoRA, which may not have LoRA weights for all layers. For example, only layers 5, 10, 15, and 20 might have corresponding LoRA weights.
It should already be possible to use adapters which do not modify every layer - if that is not the case, it is a bug and should be fixed.
lora finetune for Llama-3 has the same problem, however same code finetune Qwen1.5-7B-Chat, with vocab size 151643, has no such problem, why?
@Yard1 I've tried llama3 on v0.4.0.post1, but this issue is still present when initializing the engine with lora adapters. The latest main code seems to get further in the initialization process (ie. no explicit crash about vocab size) but seems to stall out later in the pipeline - haven't gotten a chance to debug that yet.
But to confirm - llama3 with adapters is supposed to work on 0.4.0.post1, or were you referring to the 0.4.1 prerelease?
It should be 0.4.1 - this is the commit: https://github.com/vllm-project/vllm/commit/1e96c3341a4e055ae392085fecc7a672295b71c2
@Techinix You can either manually install the wheel from the Release page or build yourself locally from the tagged git version. On my setup, local building takes around 15-30min.
For the LLaMA 3 model:
Please use the following codes for your tokenizer:
tokenizer.pad_token = tokenizer.eos_token
In this way, the vocabulary size is still 128256 .
However, if you use
tokenizer.add_special_tokens(
{'pad_token': '[PAD]'}
)
# Resize input token embeddings matrix of the model if new_num_tokens != config.vocab_size.
model.resize_token_embeddings(len(tokenizer))
The vocabulary size is increased to 128768 , which leads to this ValueError: When using LoRA, vocab size must be 32000 > vocab_size < 128512 error.
For now, I don't know the reason. If you know the reason, please leave a message. Thank you very much in advance!
Shuyue May 11th, 2024
Wouldn't it make more sense to check which layers the LoRA-adapter actually applies to? Most LoRA-frameworks don't even touch the embedding layer by default.
