Simo Ryu

Well one thing you can do is to return custom names per data in the Dreambooth dataloader, without prior preservation.

Sorry for not checking in sooner. @JunMa11 , I've implmented this like such : https://github.com/cloneofsimo/lora/blob/27145c3bd02f1240ab10de6a8c00fc37c6fcadc2/lora_diffusion/dataset.py#L187

Did you also patch the text encoder as well?

Have a look at: https://github.com/cloneofsimo/lora/blob/master/scripts/run_with_text_lora_also.ipynb

Nice to see that it works. I've found that when it overfits, tuning down both unet and text encoder helps, but with different scale. Also try learning it with prior...

https://github.com/cloneofsimo/lora#what-happens-to-text-encoder-lora-and-unet-lora Have a look at this part as well

I really haven't experimented with LoRA to conclude anything, but for me, using lr=1e-4, prior_preservation = False worked OK. here are 10 LoRA results from training 10 illustration images with...

Now you can also fine-tune CLIP encoder with LoRA as well, just like dreambooth option Checkout fine-tuning with this shell-code: https://github.com/cloneofsimo/lora/blob/master/run_lora_db_w_text.sh Checkout using LoRAs this notebook (on example LoRA i've...

You can run: https://github.com/cloneofsimo/lora/blob/master/run_lora_db_w_text.sh with few parameters changed. defaults worked OK for me, and I even stopped at 2500 steps because it was overfitting.

Nice work @pedrogengo! I'll update the readme. Meanwhile it would be nice if there was also text encoder training. Would you like to update it?