Support for multiple fine-tuned concepts & classes on one model with Dreambooth

[jslegers]

Is your feature request related to a problem? Please describe. I'd like to add multiple people and/or objects to a single model and finetune my model for that. For example, I'd like to add an entire family, together with their pets and their car, and this with as little degradation of my overall model as humanly possible.

Describe the solution you'd like I'd like to add multiple people and/or objects to a model and finetune my model with a minimal amount of degradation.

Describe alternatives you've considered I've been tinkering around with different classes and concepts, but thusfar any attempt to use a different class or concept thusfar always lead to extreme degradation.

Additional context Apparently someone already forked JoePenna/Dreambooth-Stable-Diffusion at kanewallmann/Dreambooth-Stable-Diffusion to add support for multiple concepts & classes on one model. However, that version of Dreambooth SD requires at least 24GB VRAM, which means I can't run it on either my machine or a free Google Colab, which is what I've been using so far.

Related :

• https://github.com/ShivamShrirao/diffusers/issues/21
• https://github.com/JoePenna/Dreambooth-Stable-Diffusion/issues/51

[patil-suraj]

Thanks a lot for the issue!

For now we want to keep the script simple so users can extend it on their own, also quite a few users have been reporting issues with dreambooth dreaming so I would like to not add anything extra until those issues are resolved. But happy to help you to give some pointers on how to modify the script so we can add multiple concepts.

[davidfant]

@patil-suraj possible to give pointers on how to add multiple concepts? I've been using https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth.py so far

[patil-suraj]

Sorry to be so late here, don't have much time to work on it at the moment, but will revisit next week. But if someone wants to open a PR to add this, would be happy to help :)

[dahifi]

I'm testing this as we speak and very interested.

I'm using Dreambooth on Colab, training my first model and trying to figure out how to add to it. Seems like might involve generating checkpoint files and using that as the initial model for the next pass?

Right now I'm just re-running my first two attempts in a single concepts list, but I'll definitely need to be able to refine the model later as we add new production assets to our library.

[markrmiller]

I don’t think you can currently do it, despite some projects trying to support it. It writes over the same weights or something - the two human subjects are under a lot of pressure to be merged into one. Even prior preservation does little to prevent it. Even prior preservation and lots of images with both subjects on top of the individual images can’t do enough.

At this point I think the best approach is probably multiple individually trained Lora’s or a mix of base model / Lora / textual inversion.

[Sandeep-Narahari]

Hello Team! Do we have any update on this issue, me also trying to find to train multiple classes of images , ex: person!, person2, person3 images

[nickkolok]

Hi all, I am interested in training multiple concepts, too; however, I haven't tried it yet. But ShivamShrirao's fork allows to train multiple concepts, so it looks doable!

[yiyixuxu]

cc @sayakpaul are we interested to support this?

[sayakpaul]

I am going to relay what @patil-suraj said here: https://github.com/huggingface/diffusers/issues/752#issuecomment-1271336649

But happy for anyone in case they're willing open it up to the community examples.

[nickkolok]

Do you mean that adapting ShivamShrirao's script for the newest version of diffusers and putting it to the community folder will really be welcomed?

[sayakpaul]

Yeah why not. I think that could be really beneficial for the community. @patil-suraj WDYT?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.