diffusers icon indicating copy to clipboard operation
diffusers copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

LoRA for Dreambooth

Open polavishnu4444 opened this issue 2 years ago • 7 comments

Hi, Have come across LoRA for diffusion models. It might be right for diffusers to collaborate and later get to one single diffusers SDK as Dreambooth is supported here as well in the current diffusers' library. Interested to learn about your thoughts on this. 😃

Link: https://github.com/cloneofsimo/lora

polavishnu4444 avatar Dec 15 '22 12:12 polavishnu4444

Yes, very interested in adding this to diffusers. Would love to review a PR here. Note that it could be as simple to copy the correct training script from: https://huggingface.co/spaces/ysharma/Low-rank-Adaptation/tree/main into examples/lora

patrickvonplaten avatar Dec 19 '22 16:12 patrickvonplaten

Also cc @pcuenca @williamberman @patil-suraj

patrickvonplaten avatar Dec 19 '22 16:12 patrickvonplaten

LMK if anyone is interested in adding this, very happy to help :)

patil-suraj avatar Dec 19 '22 17:12 patil-suraj

@cloneofsimo

Thomas-MMJ avatar Dec 19 '22 18:12 Thomas-MMJ

Ah sorry we've actually talked about this on the slack channel as well. Just finished final exam. I'll work on it today

cloneofsimo avatar Dec 19 '22 19:12 cloneofsimo

Happy to help with it @cloneofsimo

dhruvrnaik avatar Dec 19 '22 22:12 dhruvrnaik

Interested as well if you can teach me 😃

polavishnu4444 avatar Dec 20 '22 13:12 polavishnu4444

Just for your information:

This repository includes another LoRA implementation, however I haven't tried. https://github.com/kohya-ss/sd-scripts

Description in Japanese: https://github.com/kohya-ss/sd-scripts/blob/main/train_network_README-ja.md

They stated that the main difference between the cloneofsimo's implementation at this time is more application of LoRA to MLP in Text Encoder, FFN in U-Net, and in/out projection in Transformer.

shirayu avatar Dec 30 '22 09:12 shirayu

Starting working on it: https://github.com/huggingface/diffusers/pull/1884 -> should be ready end of week :-)

patrickvonplaten avatar Jan 02 '23 12:01 patrickvonplaten

I'm sorry for not updating this earlier. And thank you @patrickvonplaten ! I just checked this PR as well : https://github.com/huggingface/diffusers/pull/1639 this would make LoRA and Paint with words implementation much easier. (But would probably need refactor as well)

I've been trying to make LoRA better before implementing here, and with contributors, we've recently made 4 major progresses and improved fidelity considerably more. I'll try my best to implement them in the examples when I find time (or anyone can jump in)

cloneofsimo avatar Jan 02 '23 14:01 cloneofsimo

@patil-suraj @patrickvonplaten I started working on the Flax version of this a while ago and got stuck. My work so far is in #1894!

yasyf avatar Jan 03 '23 06:01 yasyf

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

github-actions[bot] avatar Jan 27 '23 15:01 github-actions[bot]

LoRA is now supported cf #1884

patil-suraj avatar Jan 27 '23 17:01 patil-suraj