diffusers
diffusers copied to clipboard
huggingface
Hypernetworks
Is your feature request related to a problem? Please describe. Support for hypernetworks since they seem to yield great results. An example can be seen here: https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/modules/hypernetworks/hypernetwork.py
info: https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/2670
Describe the solution you'd like A diffusers hypernetwork implementation that is compatible with Automatic's implementation
Describe alternatives you've considered Switching back and forth between different repos, but a one-stop shop is preferred.
Very much agree - we should definitely add this. In general there are multiple features that come out recently that require to modify the attentions for each layer - maybe we could support all of them in one go:
- https://github.com/cloneofsimo/paint-with-words-sd
- Hypernetworks
- am I missing one more @patil-suraj ?
@patil-suraj maybe if you already have a good idea here on how to implement hypernetworks you could share here?
@patrickvonplaten unrelated: would be nice to have Google's Prompt2Prompt feature. https://prompt-to-prompt.github.io/
This one also https://github.com/google/prompt-to-prompt
It would cool to have these, haven't looked deeply into it, will assign some time for it soon and reply here :)
@aungkhant0911 - we're also always happy about community contributions :-)
Happy to work on prompt-to-prompt. Maybe next week.
21/11 update: Sorry, I haven't made time for this yet. Will try in the next 1-2 weeks. 14/12 update: Sorry, still can't assign a time for this.
Hey @dblunk88,
In short we would really like to support prompt-to-prompt! So far we haven't found time to dive into it.
The best approach IMO would be to add the functionality of the paper: https://github.com/google/prompt-to-prompt/tree/main by providing a nice API to change the attention_weights and then creating a prompt-to-prompt example pipeline. Would this be interesting for you maybe?
We'll eventually look into this, but I can currently not say whether it will take 1 or 4 weeks - it depends also a bit on how much the community asks for it
Hi @patrickvonplaten . Are you planning to implement only prompt-to-prompt or general method for modifying layer? I think https://github.com/cloneofsimo/paint-with-words-sd is also very cool feature and does not require much of modification.
Yes in a first step the plan was to look into prompt-to-prompt. I'm a bit behind on this. Hoping to get to it by next week
Not yet. We recently added a feature to be able to easily tweak cross attention (cf #1639), which would make it easy to implement. Let us know if anyone from the community is interested in this, happy to help :)
We're currently adding LoRA: https://github.com/huggingface/diffusers/pull/1884 which should be very similar to how hypernetworks are done.
