[Feature Request] LoRa fine-tuning?

[zhzLuke96]

I propose to add support for LoRa fine-tuning of the model, as it has been shown to be effective for GPT models based on community feedback. Additionally, there have been attempts by the nanoGPT community to use LoRa for fine-tuning. Supporting LoRa fine-tuning could potentially take this project even further.

Adding support for LoRa fine-tuning of the model would provide users with the ability to enhance the stability of the model on their own (as currently, when history_prompt is not set, speaker selection in each generated result is random.) supporting LoRa fine-tuning may enable this project to go even further.

---

refs:

• https://github.com/cccntu/minLoRA
• https://github.com/danielgrittner/nanoGPT-LoRA

[gkucsko]

would be great. open to explore if the community can help. what's the usual speedup / memory footprint reduction that people tend to see?

[gkucsko]

closing for now, feel free to reopen if someone has time to work on it

[Leoputera2407]

Hey @gkucsko, I've tried intergrating LoRa using this repo (https://github.com/cccntu/LoRAnanoGPT/pull/1/files). I think I got the model right, however, I noticed that this repo don't have the trainer.py code out. I'm trying to adapt the trainer.py from AudioLM, since I noticed your architecture is very similar to their (except you're using Encodec, and GPT instead of T5). Meanwhile, could you share the trainer.py code?

[halilergul1]

Hi @Leoputera2407,

How did your LoRa integration perform :) ? I am also planning to delve into fine-tuning Bark. But now wondering and researching where to start. Any clue would be great.