diffusers
diffusers copied to clipboard
huggingface
Stable Diffusion 2
Model/Pipeline/Scheduler description
Perhaps you already have something in the works for this, but I couldn't see an existing issue for this.
Stable Diffusion 2 just got released on a separate repo. There are a few variations including the base, depth2img, and inpainting models. Though useable in the stable diffusion repo now, it would be really awesome to have support in diffusers to allow for ease of us in other applications!
Open source status
- [X] The model implementation is available
- [X] The model weights are available (Only relevant if addition is not a scheduler).
Provide useful links for the implementation
Model code and inference scripts can be found here
Model weights seem already available on 🤗 Hub at stabilityai/stable-diffusion-2-***
Semi-Related:
- https://github.com/Stability-AI/stablediffusion/issues/4
- https://github.com/Stability-AI/stablediffusion/issues/9
- https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/5009 [dupe]
- https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/5011
- https://github.com/db0/AI-Horde/issues/86
- https://github.com/JoePenna/Dreambooth-Stable-Diffusion/issues/112
- https://github.com/TheLastBen/fast-stable-diffusion/issues/599
- https://github.com/ShivamShrirao/diffusers/issues/143
- https://github.com/Sygil-Dev/nataili/issues/67
- https://github.com/Sygil-Dev/sygil-webui/issues/1686
rudimentary support for stable diffusion 2.0
https://github.com/MrCheeze/stable-diffusion-webui/commit/069591b06bbbdb21624d489f3723b5f19468888d
Originally posted by @152334H in https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/5011#issuecomment-1325971596
https://github.com/hafriedlander/diffusers/blob/stable_diffusion_2/scripts/convert_original_stable_diffusion_to_diffusers.py
Notes:
- Only tested on the two txt2img models, not inpaint / depth2img / upscaling
- You will need to change your text embedding to use the penultimate layer too
- It spits out a bunch of warnings about vision_model, but that's fine
- I have no idea if this is right or not. It generates images, no guarantee beyond that. (Hence no PR - if you're patient, I'm sure the Diffusers team will do a better job than I have)
Originally posted by @hafriedlander in https://github.com/huggingface/diffusers/issues/1388#issuecomment-1326135768
Here's an example of accessing the penultimate text embedding layer https://github.com/hafriedlander/stable-diffusion-grpcserver/blob/b34bb27cf30940f6a6a41f4b77c5b77bea11fd76/sdgrpcserver/pipeline/text_embedding/basic_text_embedding.py#L33
Originally posted by @hafriedlander in https://github.com/huggingface/diffusers/issues/1388#issuecomment-1326166368
doesn't seem to work for me on the 768-v model using the v2 config for v
TypeError: EulerDiscreteScheduler.init() got an unexpected keyword argument 'prediction_type'
Originally posted by @devilismyfriend in https://github.com/huggingface/diffusers/issues/1388#issuecomment-1326220609
You need to use the absolute latest Diffusers and merge this PR (or use my branch which has it in it) https://github.com/huggingface/diffusers/pull/1386
Originally posted by @hafriedlander in https://github.com/huggingface/diffusers/issues/1388#issuecomment-1326243809
(My branch is at https://github.com/hafriedlander/diffusers/tree/stable_diffusion_2)
Originally posted by @hafriedlander in https://github.com/huggingface/diffusers/issues/1388#issuecomment-1326245339
🤗 Diffusers with Stable Diffusion 2 is live!
anton-l commented (https://github.com/huggingface/diffusers/issues/1388#issuecomment-1327731012)
diffusers==0.9.0 with Stable Diffusion 2 are live! https://github.com/huggingface/diffusers/releases/tag/v0.9.0
📰 News
-
Weights for the 768x768 model is up! - Averad
- https://huggingface.co/stabilityai/stable-diffusion-2
-
#1386 - by patil-suraj and patrickvonplaten has been merged into --main by patil-suraj
-
Weights for the 512x512 models are up! - patrickvonplaten
- https://huggingface.co/stabilityai/stable-diffusion-2-base
- https://huggingface.co/stabilityai/stable-diffusion-2-inpainting
✏️ Notes & Information
Related huggingface/diffusers Pull Requests:
- #1424
- #1421
- #1420
- #1419
- #1416
- #1415
- #1414
- #1412
- #1411
- #1410
- #1402
- #1400
- #1397
- #1396
- #1386 - by patil-suraj and patrickvonplaten
👇 Quick Links:
- https://github.com/Stability-AI/stablediffusion
- https://huggingface.co/stabilityai/stable-diffusion-2
- https://huggingface.co/stabilityai/stable-diffusion-2-base
- https://huggingface.co/stabilityai/stable-diffusion-2-depth
- https://huggingface.co/stabilityai/stable-diffusion-2-inpainting
- https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler
👁️ User Submitted Resources:
- Test Version of convert_original_stable_diffusion_to_diffusers.py by hafriedlander
- Example of accessing the penultimate text embedding layer by hafriedlander
- NovelAI Improvements on Stable Diffusion
💭 User Story
Stable Diffusion 2.0 has recently been released. When you run convert_original_stable_diffusion_to_diffusers.py on the new Stability-AI/stablediffusion models the following errors occur.
convert_original_stable_diffusion_to_diffusers.py --checkpoint_path="./512-inpainting-ema.ckpt" --dump_path="./512-inpainting-ema_diffusers"
Output:
Traceback (most recent call last):
File "convert_original_stable_diffusion_to_diffusers.py", line 720, in <module>
unet.load_state_dict(converted_unet_checkpoint)
File "lib\site-packages\torch\nn\modules\module.py", line 1667, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel:
size mismatch for down_blocks.0.attentions.0.proj_in.weight: copying a param with shape torch.Size([320, 320]) from checkpoint, the shape in current model is torch.Size([320, 320, 1, 1]).
size mismatch for down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([320, 768]).
size mismatch for down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([320, 768]).
size mismatch for down_blocks.0.attentions.0.proj_out.weight: copying a param with shape torch.Size([320, 320]) from checkpoint, the shape in current model is torch.Size([320, 320, 1, 1]).
.... blocks.1.attentions blocks.2.attentions etc. etc.
testing in progress on the horde https://github.com/Sygil-Dev/nataili/tree/v2 try it out Stable Diffusion 2.0 on our UI's
https://tinybots.net/artbot https://aqualxx.github.io/stable-ui/ https://dbzer0.itch.io/lucid-creations
https://sigmoid.social/@stablehorde/109398715339480426
SD 2.0
- [x] Initial implementation ready for testing
- [ ] img2img
- [ ] inpainting
- [ ] k_diffusers support
Originally posted by @AlRlC in https://github.com/Sygil-Dev/nataili/issues/67#issuecomment-1326385645
- https://github.com/TheLastBen/fast-stable-diffusion/commit/11fd38bfbd2f1ed42449b37ba88ba324ff42ba43
Create pathsV2.py
- https://github.com/TheLastBen/fast-stable-diffusion/commit/fe445d986f08a1134f26f5efcd1c0829f34bc481
Support for SD V.2
- https://github.com/TheLastBen/fast-stable-diffusion/commit/da9b38010c2edc8fcccf2b0b70f321af30c0ecb8
fix
- https://github.com/TheLastBen/fast-stable-diffusion/commit/6c84728c72bd9735b0a5be4c62a292554c3b41d1
fix
- https://github.com/TheLastBen/fast-stable-diffusion/commit/04ba92b1931ab6aa0269a0516640f8874b004885
fix
- https://github.com/TheLastBen/fast-stable-diffusion/commit/ebea13401da873b3420fdf6f0fa02df567534a55
Create sd_hijackV2.py
- https://github.com/TheLastBen/fast-stable-diffusion/commit/88496f5199c82e9c5ee2ae40bc980140d8cd4ce5
Create sd_samplersV2.py
- https://github.com/TheLastBen/fast-stable-diffusion/commit/f324b3d85473d308ebeefb03de58ae6eb9070f42
fix V2
Originally posted by @0xdevalias in https://github.com/TheLastBen/fast-stable-diffusion/issues/599#issuecomment-1326446674
Should work now, make sure you check the box "redownload original model" when choosing V2
https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast_stable_diffusion_AUTOMATIC1111.ipynb
Requires more than 12GB of RAM for now, so free colab probably won't suffice.
Originally posted by @TheLastBen in https://github.com/TheLastBen/fast-stable-diffusion/issues/599#issuecomment-1326461962
From @pcuenca on the HF discord:
We are busy preparing a new release of
diffusersto fully support Stable Diffusion 2. We are still ironing things out, but the basics already work from the main branch in github. Here's how to do it:
- Install diffusers from github alongside its dependencies:
pip install --upgrade git+https://github.com/huggingface/diffusers.git transformers accelerate scipy
- Use the code in this script to run your predictions:
from diffusers import DiffusionPipeline, EulerDiscreteScheduler import torch repo_id = "stabilityai/stable-diffusion-2" device = "cuda" scheduler = EulerDiscreteScheduler.from_pretrained(repo_id, subfolder="scheduler", prediction_type="v_prediction") pipe = DiffusionPipeline.from_pretrained(repo_id, torch_dtype=torch.float16, revision="fp16", scheduler=scheduler) pipe = pipe.to(device) prompt = "High quality photo of an astronaut riding a horse in space" image = pipe(prompt, width=768, height=768, guidance_scale=9).images[0] image.save("astronaut.png")
More related:
- https://github.com/huggingface/diffusers/issues/1398
- https://github.com/huggingface/diffusers/issues/1404
how sure are you that your conversion is correct? I'm trying to diagnose a difference I get between your 768 weights and my conversion script. There's a big difference, and in general I much prefer the results from my conversion. It seems specific to the unet - if I replace my unet with yours I get the same results.
Originally posted by @hafriedlander in https://github.com/huggingface/diffusers/issues/1388#issuecomment-1327018829
OK, differential diagnostic done, it's the Tokenizer. How did you create the Tokenizer at https://huggingface.co/stabilityai/stable-diffusion-2/tree/main/tokenizer? I just built a Tokenizer using
AutoTokenizer.from_pretrained("laion/CLIP-ViT-H-14-laion2B-s32B-b79K")- it seems to give much better results.Originally posted by @hafriedlander in https://github.com/huggingface/diffusers/issues/1388#issuecomment-1327031107
I've put "my" version of the Tokenizer at https://huggingface.co/halffried/sd2-laion-clipH14-tokenizer/tree/main. You can just replace the tokenizer in any pipeline to test it if you're interested.
Originally posted by @hafriedlander in https://github.com/huggingface/diffusers/issues/1388#issuecomment-1327077503
diffusers==0.9.0with Stable Diffusion 2 is live!https://github.com/huggingface/diffusers/releases/tag/v0.9.0
Originally posted by @anton-l in https://github.com/huggingface/diffusers/issues/1388#issuecomment-1327731012
when will Dreambooth support sd2
While it's not dreambooth, this repo seems to have support for finetuning SDv2:
- https://github.com/smirkingface/stable-diffusion
- https://github.com/smirkingface/stable-diffusion#news
Added support for inference and finetuning with the SD 2.0 base model (inpainting is still unsupported).
- https://github.com/smirkingface/stable-diffusion/blob/main/docs/sd2.0.md
Originally posted by @0xdevalias in https://github.com/JoePenna/Dreambooth-Stable-Diffusion/issues/112#issuecomment-1327993709
And looking at the huggingface/diffusers repo, there are a few issues that seem to imply people may be getting dreambooth things working with that (or at least trying to), eg.:
- https://github.com/huggingface/diffusers/issues/1429
Originally posted by @0xdevalias in https://github.com/JoePenna/Dreambooth-Stable-Diffusion/issues/112#issuecomment-1327998619