Benjamin Bossan

> If I want to implement what I said in question 2, what should I do? It depends on how far you want to take this. If I wanted to...

Thanks @huismiling, could you please run `make style` on this PR?

One possibility that you could try is to not add the embeddings to `modules_to_save` but instead just LoRA-tune them by adding them to `target_modules`. This could be especially useful for...

This probably means that your PEFT installation is broken. Can you import anything else successfully from PEFT?

Load the model from transformers with `quantization_config=HqqConfig(...)`, the rest is the same. Here is an example: https://github.com/huggingface/peft/blob/fb7f2796e5411ee86588447947d1fdd5b6395cad/tests/test_gpu_examples.py#L2386C28-L2428

**Update**: Unfortunately, I could not get past the last few issues mentioned above. Therefore, I'll close this PR in favor of #1806. That PR has more code changes, so it...

Ping @yfeng95, @Zeju1997, @YuliangXiu

_not sure why you closed the issue but I can replicate it_ Thanks for bringing this to our attention. Indeed, the `lora_E` parameter should be initialized as zeros but it...

> Thanks for your response. I closed this issue since the difference in losses is due to the orthogonal regularization term. I see, thanks for explaining. Let's still keep this...

I can replicate the results. When it comes to this message > Loading adapter weights from None led to unexpected keys not found in the model: ['text_projection.lora.down.weight', 'text_projection.lora.up.weight']. at first...