stable-diffusion-webui
stable-diffusion-webui copied to clipboard
AUTOMATIC1111
[Feature Request] Implement DreamBooth with memory optimizations. This is much better textual inversion via fine-tuning
As many have notes, the default textual inversion doesn't work very well but the new DreamBooth approach works great. See here for setting it up with the memory optimizations: https://github.com/gammagec/Dreambooth-SD-optimized and here for more info: https://www.reddit.com/r/StableDiffusion/comments/xmkwmp/i_got_dreambooth_for_sd_to_work_on_my_3090_w24_gb/
Another example of how well it works: https://www.reddit.com/r/StableDiffusion/comments/xn1jln/i_used_googles_dreambooth_to_finetune_the/
This would be great to have native in AUTOMATIC1111 to make it more accessible.
we have similar issue open here: #914
Thank you, I missed that one. They didn't seem to mention the new memory optimizations so I'll add a comment there.
This would be great to have native in AUTOMATIC1111 to make it more accessible.
Dreambooth requires just under 24GB with all optimizations. Ease of use isn't the problem, the raw hardware requirements are. This doesn't belong in user-facing UI.
This would be great to have native in AUTOMATIC1111 to make it more accessible.
Dreambooth requires just under 24GB with all optimizations. Ease of use isn't the problem, the raw hardware requirements are. This doesn't belong in user-facing UI.
Someone already has it working in 17.7GB of VRAM, and under 16GB sounds very plausible to implement, meaning it can run on a 3060 16GB card which is very affordable. See here: https://github.com/huggingface/diffusers/pull/554#issuecomment-1258751183
It's now down to 12.5GB (but for diffusers, so not usable with this repo anyway). Yet I'd still argue that training a model is an entirely separate use case from using a model and should remain its own project, rather than vendoring an entire new codebase under this one
@hopibel I would say that training a model for personalized styles or objects or people is one way many people want to use the model, so perhaps it should be included, if memory optimizations can allow it to work on a ~12GB card.
dreambooth training script was merged to diffusers https://github.com/huggingface/diffusers/tree/main/examples/dreambooth
Then it seems it can be added to Automatic1111? (which then can be used in a colab?)
Models in the diffusers format aren't currently supported afaik. Either support would have to be added or the dreambooth optimizations would have to be ported to the CompVis-based implementation
colab for training dreambooth with diffusers is out: https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_dreambooth_training.ipynb, includes a gradio web ui, from the gradio side it can be added to this web ui but would be easier it is uses diffusers
Apparently the Dreambooth fork below can produce a ckpt file that can be used with this repo just like any other ckpt model file, along with your custom token (the ckpt is only 2GB too, which is odd, but they say it includes the full v1.4 SD model still, just removes cruft...): https://github.com/gammagec/stable-dreambooth-optimized
Unfortunately, still requires 24GB of VRAM for the training bit, but if some other forks' memory optimizations were integrated, that might not be a problem. Lowest I've seen so far for training is 12.5GB...
Someone's got it working now on 11GB of VRAM, a RTX3060!
https://www.reddit.com/r/StableDiffusion/comments/xsrafl/dreambooth_in_11gb_of_vram/?utm_medium=android_app&utm_source=share
Let's add a Dreambooth training tab to the UI to create personalized ckpt models! How can I help?
Down to 10GB now! https://www.reddit.com/r/StableDiffusion/comments/xtc25y/dreambooth_stable_diffusion_training_in_10_gb/
Hmm, might not output ckpt model though...
There's a new script to do this now https://www.reddit.com/r/StableDiffusion/comments/xu38ao/now_possible_to_use_dreambooth_colab_models_in/
https://github.com/d8ahazard/sd_dreambooth_extension requires at least 10gb, available as an extension.
