Everything is Working Great, quick question....

[MushroomFleet]

as seen here, (example outputs included) https://twitter.com/MushroomFleet/status/1602341447952437249 and below (just the merge experiments)

What are your advised Training settings? I did use a bunch that i used in other Dreambooth methods, but it would be interesting to know if you have a recommended setting for LORA because it is a little different.

Thanks for your hard work in bringing this to us :)

[cloneofsimo]

I really haven't experimented with LoRA to conclude anything, but for me, using lr=1e-4, prior_preservation = False worked OK. here are 10 LoRA results from training 10 illustration images with 30000 steps.

[nitrosocke]

I haven't done any style training with it (yeah I know, how is this possible?..) I used it mainly for subject training and refining a few celebrities in base SD, here are my results in conjunction with my dreambooth models:

The settings I used where the following: --pretrained_model_name_or_path=$MODEL_NAME --weight_name=$WEIGHT_NAME --instance_data_dir=$INSTANCE_DIR --output_dir=$OUTPUT_DIR --instance_prompt="[celeb name]" --resolution=768 --train_batch_size=4 --mixed_precision="fp16" --use_8bit_adam --gradient_accumulation_steps=1 --learning_rate=1e-4 --lr_scheduler="constant" --save_steps=200 --max_train_steps=1000

[pedrogengo]

@nitrosocke Do you mind to share the process you did to get these results? Im trying to use only LORA to finetune SD, but the results were not that good.

[nitrosocke]

@pedrogengo Sure thing here is my workflow:

• Choose 5-10 images of a person
• Crop/resize to 768x768 for SD 2.1 training
• Following settings worked for me: train_batch_size=4, mixed_precision="fp16", use_8bit_adam, learning_rate=1e-4, lr_scheduler="constant", save_steps=200, max_train_steps=1000
• for systems with less VRAM than 24GB use a lower train_batch_size like 2 or 1.
• make sure to use xformers and 8bit adam
• for subjects already know to SD images*100 worked great, for subjects unknown to SD more steps or a higher LR are required
• training on a 3090 takes ~20 min for 1k steps

[pedrogengo]

@nitrosocke Did you try with some unknown subjects? My problem is when Im trying to finetune one with images from myself (as you may know, Im not a celeb haha)

And thanks for sharing! What prompt did you use to generate the last two images from your grid?

[nitrosocke]

@pedrogengo I did try training my wife and the settings didn't work. I haven't done any more testing on that but I assume it's something like 2k steps for 5 images. Would have to try though.

Prompt for Joseph was "modern disney style joseph gordon levitt", it's using my custom model and the JGL training .pt on top.

[MushroomFleet]

Thanks for all the responses everyone!!

I generally only overwrite known concepts (like actors) with photos of myself. I'll take a look at all the settings you all posted and tweak mine accordingly :)

I'm helping to test a local installed UI implementation that now included LORA dreambooth so any information is very useful for guiding users !!

[cloneofsimo]

Now you can also fine-tune CLIP encoder with LoRA as well, just like dreambooth option

Checkout fine-tuning with this shell-code: https://github.com/cloneofsimo/lora/blob/master/run_lora_db_w_text.sh

Checkout using LoRAs this notebook (on example LoRA i've made): https://github.com/cloneofsimo/lora/blob/master/scripts/run_with_text_lora_also.ipynb

[amrrs]

@cloneofsimo Thanks for the work. Do you have any Colab / Notebook for basic training. I currently saw 4 scripts on the scripts folder, but not sure if you got the example there or somewhere else.

Basically, if i've got 4 768x768 images and I want to fine-tune SD2.0, then which is the script I should look at?

[cloneofsimo]

You can run: https://github.com/cloneofsimo/lora/blob/master/run_lora_db_w_text.sh with few parameters changed. defaults worked OK for me, and I even stopped at 2500 steps because it was overfitting.

[amrrs]

Thank you @cloneofsimo sorry for a dumb question. So the input images should be inside this folder which is assigned as the instance directory INSTANCE_DIR="./data_example_text" ?

[pedrogengo]

Yes @amrrs

[pedrogengo]

@amrrs I did this Colab notebook if you want to perform all the steps on Colab:

https://colab.research.google.com/drive/1iSFDpRBKEWr2HLlz243rbym3J2X95kcy?usp=sharing

@cloneofsimo If you like, you can update the README with it :)

[Daniel-Kelvich]

@pedrogengo is it possible to run colab training with batch_size=5? It is the default setting but I'm getting CUDA_OUT_OF_MEMORY

[amerkay]

@pedrogengo is it possible to run colab training with batch_size=5? It is the default setting but I'm getting CUDA_OUT_OF_MEMORY

Which GPU are you using? T4 15GB (standard) or A100 40GB (premium)?

I'm trying to figure out ideal settings for the A100.

[pedrogengo]

Sorry, I didn't test with batch_size=5. I was using 1 during my experiments. What you can do is use 1 and the gradient_step can be 5, so you update the gradients only after 5 steps, which is the same of use batch_size=5. I will create a field for this info too on Colab

[pedrogengo]

@amerkay @Daniel-Kelvich I just updated the Colab with Gradient Accumulation Steps! Enjoy :)

[cloneofsimo]

Nice work @pedrogengo! I'll update the readme. Meanwhile it would be nice if there was also text encoder training. Would you like to update it?

[pedrogengo]

Sure thing! I can to it until EOD

[brian6091]

@pedrogengo @cloneofsimo the script train_lora_dreambooth.py seems to be missing a call for accelerate to manage the accumulation context:

https://huggingface.co/docs/accelerate/v0.13.2/en/package_reference/accelerator#accelerate.Accelerator.accumulate

so I'm not sure passing the parameter will do anything.

[cloneofsimo]

Right gradient accumulation doesn't work now because it implicitly updates all other params wrapped inside it. So i removed it.

[brian6091]

Even if you set return_grad=None or filter the parameters?

[cloneofsimo]

Yes I think so. But Im not really used to accelerate package so it was probably wasn't the way to fix it. I'll try to make it work with grad accumulation

[brian6091]

Just gonna drop a link to more training/tuning discussion here: https://github.com/cloneofsimo/lora/discussions/37

[pedrogengo]

@cloneofsimo I just updated the colab and as a workaround to gradient accumulation I'm making lr / batch_size. I know it is not the best approach but it is better than nothing haha

https://colab.research.google.com/drive/1iSFDpRBKEWr2HLlz243rbym3J2X95kcy?usp=sharing#scrollTo=ZDMXQnKfat1-

[scaraffe]

Is there a way to train new images with manually added captions?

[brian6091]

@scaraffe

You can try this colab notebook: https://github.com/brian6091/Dreambooth

that allows using captions with @cloneofsimo's lora training