Can OpenLlama model apply lora trained by llama?

[sunyuhan19981208]

Can OpenLlama model apply lora trained by llama?

[LudwigStumpp]

Hi,

let me try to understand your question:

You are asking if the Alpaca-LoRA adapter weights (e.g. https://huggingface.co/tloen/alpaca-lora-7b) that were trained on top of the non-commercial LlaMA model (e.g https://huggingface.co/decapoda-research/llama-7b-hf in the case of above adapter) are directly applicable to the OpenLLaMA model?

[sunyuhan19981208]

Hi,

let me try to understand your question:

You are asking if the Alpaca-LoRA adapter weights (e.g. https://huggingface.co/tloen/alpaca-lora-7b) that were trained on top of the non-commercial LlaMA model (e.g https://huggingface.co/decapoda-research/llama-7b-hf in the case of above adapter) are directly applicable to the OpenLLaMA model?

Yes, this is what I concerned.

[LudwigStumpp]

I understand. Based on my experience and knowledge, I think that what you propose is not possible, as the weights of the original LlaMA model and the OpenLlaMA model are different (even though their architecture and general training procedure is the same). Therefore, the adapter (with its weights) which was trained to "work well" with the original LlaMA model weights no longer works as expected for the OpenLlaMA model.

On the other hand, no one hinders you from training your own LoRA-adapter on the OpenLlaMA model. If you want to do that, I can recommend the following repository as a starting point: https://github.com/tloen/alpaca-lora.

I previously did the same for a German translation of the Alpaca dataset: https://huggingface.co/ludwigstumpp/esel-lora-7b The whole training was possible to perform in Google Colab Pro (which is 10 euros a month). Or if you have a RTX3090 or 4090 lying aroung, you can easily do it on your local machine as well.

Hope that helps and answers your question. If so, feel free to close this issue.

[Muhtasham]

Jumping on this @LudwigStumpp, did you train few epochs on pure german data before fine-tuning?

[sunyuhan19981208]

I understand. Based on my experience and knowledge, I think that what you propose is not possible, as the weights of the original LlaMA model and the OpenLlaMA model are different (even though their architecture and general training procedure is the same). Therefore, the adapter (with its weights) which was trained to "work well" with the original LlaMA model weights no longer works as expected for the OpenLlaMA model.

On the other hand, no one hinders you from training your own LoRA-adapter on the OpenLlaMA model. If you want to do that, I can recommend the following repository as a starting point: https://github.com/tloen/alpaca-lora.

I previously did the same for a German translation of the Alpaca dataset: https://huggingface.co/ludwigstumpp/esel-lora-7b The whole training was possible to perform in Google Colab Pro (which is 10 euros a month). Or if you have a RTX3090 or 4090 lying aroung, you can easily do it on your local machine as well.

Hope that helps and answers your question. If so, feel free to close this issue.

Okay, I see, I have ever apply StableDiffusion Lora which trained from one checkpoint to another checkpoint, and that worked, therefore I am thinking about whether llama loras have the same usage.

[LudwigStumpp]

Jumping on this @LudwigStumpp, did you train few epochs on pure german data before fine-tuning?

No, I did not do any intermediate pre-training on german language. I directly began with the german instruction-tuning. Maybe that might be a good idea to do. However I found out that the original Llama model is actually already able to output german responses.