Conversion Script: Update to Stable Diffusion 1.5 (EMA and non-EMA)

[redwraith2]

Since stable diffusion 1.5 was out, I was hoping to use it with the imagic colab. Could you upgrade diffusers to use the new model instead of the old one?

[samedii]

It's supported. Switch the model id to runwayml/stable-diffusion-v1-5

[patrickvonplaten]

Hey @redwraith2, it should work out of the box :-)

[InB4DevOps]

I tried to convert v1-5-pruned.ckpt (the 7gb version) to a diffuser model folder using scripts/convert_original_stable_diffusion_to_diffusers.py but the resulting model folder only has 4GB. The script also downloaded something which was only 4GB. Does the script need an update maybe?

[patrickvonplaten]

Hey @InB4DevOps,

The reason it only has 4GB is because we only extract the "non-ema" (not exponentially mean averaged) weights of the unet from the checkpoint and leave the "ema" weights - see: https://github.com/huggingface/diffusers/blob/2fb8fafa4b761f6fc144cf75a6f6f0ea6af3a1c1/scripts/convert_original_stable_diffusion_to_diffusers.py#L295 . But I agree that this is a bit badly documented at the moment! I'll open a PR :-)

[patrickvonplaten]

python scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path ../stable_diffusion_checkpoints/stable-diffusion-v1-5/v1-5-pruned.ckpt --dump_path ../ --scheduler_type lms --extract_ema

should now work with https://github.com/huggingface/diffusers/pull/991

[patrickvonplaten]

If you don't want the EMA weights you can just run:

python scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path ../stable_diffusion_checkpoints/stable-diffusion-v1-5/v1-5-pruned.ckpt --dump_path ../ --scheduler_type lms

[InB4DevOps]

Thank you @patrickvonplaten Will give this a try later!

[toyxyz]

I have a similar problem. After the diffusers update, I converted SD v15.ckpt and then converted it back to cktp, but the file size became smaller. 7gb->4gb. And when converting ckpt files of different sizes with the convert_original_stable_diffusion_to_diffusers.py script, all results are the same size. is there something i am missing?

python convert_original_stable_diffusion_to_diffusers.py --checkpoint_path C:\ckpt\v1-5-pruned.ckpt --dump_path C:\ckpt\v15_2 --scheduler_type lms --extract_ema

python convert_diffusers_to_original_stable_diffusion.py --model_path C:\ckpt\v15_2 --checkpoint_path C:\ckpt\test_v15_2.ckpt

[patrickvonplaten]

The conversion should always go from 7GB to 4GB because either the EMA or non-EMA weights are used :-)

[toyxyz]

The conversion should always go from 7GB to 4GB because either the EMA or non-EMA weights are used :-)

Oh I understand! Which one is suitable for Dreambooth, Ema or Non Ema?

[patrickvonplaten]

Good question! I think the non-ema are better, but let's ask the experts here cc @patil-suraj @pcuenca :-)

[patil-suraj]

For dreambooth I didn't observe much difference between EMA and non-EMA, so far I've only been using EMA and it works fine.

[d8ahazard]

For dreambooth I didn't observe much difference between EMA and non-EMA, so far I've only been using EMA and it works fine.

Maybe you can answer a question that's been bugging me since this was introduced.

Well, two, actually.

One: In the current iteration, the extraction for EMA says it's using one or the other. But it's actually renaming the EMA keys and adding them to the dict, then grabbing all the non-ema keys and writing them to the dict. I don't know if this overwrites anything, but the warning string says it's doing one thing, when the ema flag really just prepends those values if found.

Two: Shouldn't the StableDiffusionPipeline and conversion scripts have a separate param for the EMA values, which are then extracted and saved independently if they're found? In my dreambooth training script, we can use the EMA params, but it's still not preserving the originals.

Not sure if this even impacts training or re-training of a model, but IMHO, for completeness, I'd think we'd want to handle the EMA values separately when training and then write them back to the checkpoint where we found them, no?

[victorchall]

Just to put a bug in the team's ear and add a +1 here, I'm working on more general case fine tuning projects (as opposed to dreambooth) and it would be appreciated to have the control over both weight sets in general as I experiment with diffusers and test different techniques. Thanks!

[patrickvonplaten]

I don't fully understand the question here. Note that you can extract both EMA and non-EMA weights from the conversion script as explained here: https://github.com/huggingface/diffusers/issues/931#issuecomment-1291827684

Could you maybe open a PR to better understand what you would like to have changed?

[patrickvonplaten]

Think we can close this one now no? One can convert both EMA and non-EMA checkpoitns.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.