[FEAT] Integrate LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models

[mbaddar1]

Feature request

Paper link : https://aclanthology.org/2024.naacl-long.174.pdf Code link : https://github.com/yifanycc/loretta

Motivation : Tensor-Trains are one of the promising low-rank structures heavily applied for parameter-efficient machine learning, esp. LLMs

LoRETTA takes LoRa to the next level by re-parameterizing the updates for pretrained weights to set of two tensor-trains

Performance : The paper claims that loretta leads to significant parameters reduction wrt LoRa

Also the paper claims LoRETTA achieves comparable results to other BERT family models

Note I have contacted authors about this FR https://github.com/yifanycc/loretta/issues/7

Your contribution

If it is OK, I would love to start a PR on that

[BenjaminBossan]

Thank you for your proposal to add LoRETTA to PEFT. The paper looks interesting enough from a methodology point of view, so it could be a good fit. Before you work on the PR, let's give the authors a bit of time to respond to your request on their repo. If, in one week, we don't hear back or they give their thumbs up, you can proceed.

Just a heads up, I skimmed their repo and saw that it's based on a very old version of PEFT, so integrating LoRETTA will require a bit of work that goes beyond copy-pasting their code base.

[mbaddar1]

Thanks @BenjaminBossan. Good idea, let's give authors one week or so

[mbaddar1]

Hi @BenjaminBossan I have contacted Yifan Yang (Github, Website) via email and he showed a great interest in integrating LoRETA with PEFT. I you would like me to forward the conversation to you let me know. I will start a PR and share a development plan (As it is my first time contributing to PEFT to discuss) Thanks.

[BenjaminBossan]

@mbaddar1 There is no need to forward the conversation. If you've got the okay from Yifan Yang, just go ahead and start the PR.

I will start a PR and share a development plan (As it is my first time contributing to PEFT to discuss)

That's fine, you can start with a draft PR and we can provide early feedback.

[mbaddar1]

Hi @BenjaminBossan I just wanted a bit of a "kickstart help" Is there any simple tutorial that show a simple script using different "tuners" to get a deeper understanding of how "config" and "model" py file should work ? Any documentation to understand their design. Yes I can delve into code and debug but thought examples and documentation might help

[BenjaminBossan]

There is no specific tutorial for adding new PEFT methods. I think it's easiest if you check one of the past additions, e.g. #2678 is a recent one. As a first step, you can focus on the core implementation and skip docs, examples and tests, and you will see it's not actually that much code. If there are any questions, feel free to ask them.

[mbaddar1]

Thanks , I will go thru this PR as a learning starter. If I got stuck will let you know

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[mbaddar1]

Commenting to keep alive