Google Prompt to Prompt

[bbecausereasonss]

Would be awesome to have integrated!

https://github.com/google/prompt-to-prompt

[horribleCodes]

This code was tested with Python 3.8, Pytorch 1.11 using pre-trained models through huggingface / diffusers.

This fork is only tested on Python 3.10.6, hard to say if there won't be major problems converting to a different version.

The code was tested on a Tesla V100 16GB but should work on other cards with at least 12GB VRAM.

I'd wager that only a small minority has access to that kind of VRAM - I personally only have 8GB, and certainly not a Tesla. Unless there's an adaptation that works with lesser hardware, we probably won't see this anytime soon.

[nulqwy]

wait, i just realized, isnt "img2img alternative test" does the same thing? (i watched 12 minute paper about this thing, cool thing)

[bbecausereasonss]

No alternative test is VERY janky, hardly works anymore.

[nulqwy]

ok, response was fast, well, still, i have only 6 GB of vram, so, maybe idc

[cvar66]

If it runs on 12 GB now, it would probably run on 8 GB with all the optimizations in this fork.

[bbecausereasonss]

FYI this is needed to make it work >

https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_stable_diffusion_to_diffusers.py

[Tophness]

Pretty good demo here: https://www.youtube.com/watch?v=lHcPtbZ0Mnc Seems like it could be quite a bit better than the other cross-attention control methods / prompt editing etc. I think at this point we might as well have a drop down list for cross attention control types though because we need to do more experimentation

[Tophness]

FYI this is needed to make it work >

https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_stable_diffusion_to_diffusers.py

Most modules made for diffusers tend to get ported to standard SD rather than having to do it the other way around

[checksummaster]

FYI this is needed to make it work > https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_stable_diffusion_to_diffusers.py

Most modules made for diffusers tend to get ported to standard SD rather than having to do it the other way around

It needed to convert ckpt to run with the current state of prompt to prompt ... I already tested it with ckpt done with some dreamboot I have in my automatic1111 (I was not keeping the original lol, then I needed to convert back)

I have a colab, will share it after I clean it.

[checksummaster]

Small note... If you want to try it, it run well on cpu, just take 10minutes per image but .... Just force device="cpu" in the code.

[checksummaster]

colab: https://colab.research.google.com/drive/1wLKjM5NFGXlRsQ0_oevhxddeoCNE7eIU?usp=sharing

[ghost]

I'm not sure how this can adapted, but there's https://github.com/cccntu/efficient-prompt-to-prompt which is supposedly a much faster implementation of the same idea

[checksummaster]

One easy way is

1. converting a ckpt using and convert it to a diffusion model using https://raw.githubusercontent.com/huggingface/diffusers/main/scripts/convert_original_stable_diffusion_to_diffusers.py
2. call the code as is (the original code uses the diffusion model) (see my collab that does 1 and 2)
3. return the result to the interface

The long way is rewriting code to deal with ckpt ... Should be easy for a PyTorch guru ....