RunwayML inpainting checkpoint support

[DenkingOfficial]

Is your feature request related to a problem? Please describe. No, thats not a problem, but huge enhancement.

Describe the solution you'd like I'd like to use inpainting model by RunwayML in Webui, for now the model is incompatible. All source code and a link to the chechpoint available on RunwayML Stable Diffusion repo:

https://github.com/runwayml/stable-diffusion

Check the results, they are amazing:

Nowadays webui throws an error while loading this:

RuntimeError: Error(s) in loading state_dict for LatentDiffusion:
        size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 9, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).

Describe alternatives you've considered Please add compatibility with models which shape is different than regular SD models. It sounds like a huge work but I'm hoping that it's possible.

Additional context AUTOMATIC1111 and Contributors thank you so much for this work!!! Love you <3

[nagolinc]

I started creating a script (https://gist.github.com/nagolinc/864b142c2cd8ca613a508d29e3d23a17) to do inpainting based off of the RunwayML repo. But my computer runs out of VRAM when I try to use it.

If someone with an RTX3090 wants to try, copy that script into /stable-diffusion-webui/scripts and make sure to set the environment variable STABLE_DIFFUSION_COMMIT_HASH=5456f8adb684f6bac3c29d6808b5c1341176c0c4

I might try setting it up on Colab later just to verify the script is actually working (which it probably isn't).

[nicolai256]

I think this has to do with the config inference file you're using, try using the one from the inpaint repo?

[1blackbar]

So if that one works and its being used with interchangeably LaMa inpainting weights, does it mean it is possible to use lama weights in SD webui to inpaint ? I know that you can run SD weights in LaMa already. https://github.com/Sanster/lama-cleaner

[batrlatom]

I started creating a script (https://gist.github.com/nagolinc/864b142c2cd8ca613a508d29e3d23a17) to do inpainting based off of the RunwayML repo. But my computer runs out of VRAM when I try to use it.

If someone with an RTX3090 wants to try, copy that script into /stable-diffusion-webui/scripts and make sure to set the environment variable STABLE_DIFFUSION_COMMIT_HASH=5456f8adb684f6bac3c29d6808b5c1341176c0c4

I might try setting it up on Colab later just to verify the script is actually working (which it probably isn't).

I have 3090, will try

EDIT: what am I missing?

Checking out commit for Stable Diffusion with hash: 5456f8adb684f6bac3c29d6808b5c1341176c0c4...
Traceback (most recent call last):
  File "launch.py", line 199, in <module>
    prepare_enviroment()
  File "launch.py", line 173, in prepare_enviroment
    git_clone(stable_diffusion_repo, repo_dir('stable-diffusion'), "Stable Diffusion", stable_diffusion_commit_hash)
  File "launch.py", line 81, in git_clone
    run(f'"{git}" -C {dir} checkout {commithash}', f"Checking out commit for {name} with hash: {commithash}...", f"Couldn't checkout commit {commithash} for {name}")
  File "launch.py", line 33, in run
    raise RuntimeError(message)
RuntimeError: Couldn't checkout commit 5456f8adb684f6bac3c29d6808b5c1341176c0c4 for Stable Diffusion.
Command: "git" -C repositories/stable-diffusion checkout 5456f8adb684f6bac3c29d6808b5c1341176c0c4
Error code: 128
stdout: <empty>
stderr: fatal: reference is not a tree: 5456f8adb684f6bac3c29d6808b5c1341176c0c4

[nagolinc]

I started creating a script (https://gist.github.com/nagolinc/864b142c2cd8ca613a508d29e3d23a17) to do inpainting based off of the RunwayML repo. But my computer runs out of VRAM when I try to use it. If someone with an RTX3090 wants to try, copy that script into /stable-diffusion-webui/scripts and make sure to set the environment variable STABLE_DIFFUSION_COMMIT_HASH=5456f8adb684f6bac3c29d6808b5c1341176c0c4 I might try setting it up on Colab later just to verify the script is actually working (which it probably isn't).

I have 3090, will try

EDIT: what am I missing?

Checking out commit for Stable Diffusion with hash: 5456f8adb684f6bac3c29d6808b5c1341176c0c4...
Traceback (most recent call last):
  File "launch.py", line 199, in <module>
    prepare_enviroment()
  File "launch.py", line 173, in prepare_enviroment
    git_clone(stable_diffusion_repo, repo_dir('stable-diffusion'), "Stable Diffusion", stable_diffusion_commit_hash)
  File "launch.py", line 81, in git_clone
    run(f'"{git}" -C {dir} checkout {commithash}', f"Checking out commit for {name} with hash: {commithash}...", f"Couldn't checkout commit {commithash} for {name}")
  File "launch.py", line 33, in run
    raise RuntimeError(message)
RuntimeError: Couldn't checkout commit 5456f8adb684f6bac3c29d6808b5c1341176c0c4 for Stable Diffusion.
Command: "git" -C repositories/stable-diffusion checkout 5456f8adb684f6bac3c29d6808b5c1341176c0c4
Error code: 128
stdout: <empty>
stderr: fatal: reference is not a tree: 5456f8adb684f6bac3c29d6808b5c1341176c0c4

cd {automatic}/repositories/stable-diffusion git remote add https://github.com/runwayml/stable-diffusion.git git fetch

[bbecausereasonss]

Any news? :)

[willlllllio]

@nagolinc after half()-ing the model the streamlit demo it includes runs successfully on a 6GB card for me so shouldn't need anything near a 3090.

[nagolinc]

@willlllllio Yeah, my implementation was just inefficent (loading full model in addition to already loaded model)

[nagolinc]

@bbecausereasonss Looks like @random-thoughtss has created a working implementation here!

https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/3192

[santiagopoli]

I know this was supposedly fixed, but I'm still facing the error with the latest version of both https://github.com/runwayml/stable-diffusion and https://github.com/AUTOMATIC1111/stable-diffusion-webui

Logs:

Loading weights [3e16efc8] from /home/stable-diffusion-webui/models/Stable-diffusion/model-1-5.ckpt
Global Step: 440000
Downloading: 100%|██████████| 1.59G/1.59G [00:16<00:00, 105MB/s]
Traceback (most recent call last):
  File "launch.py", line 206, in <module>
    start_webui()
  File "launch.py", line 201, in start_webui
    webui.webui()
  File "/home/stable-diffusion-webui/webui.py", line 124, in webui
    initialize()
  File "/home/stable-diffusion-webui/webui.py", line 85, in initialize
    modules.sd_models.load_model()
  File "/home/stable-diffusion-webui/modules/sd_models.py", line 231, in load_model
    load_model_weights(sd_model, checkpoint_info)
  File "/home/stable-diffusion-webui/modules/sd_models.py", line 173, in load_model_weights
    missing, extra = model.load_state_dict(sd, strict=False)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1604, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LatentDiffusion:
	size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 9, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).

(Running in GCP on an ubuntu-based docker container. GPU is Nvidia T4)

[zer0TF]

I am having the same issue as @santiagopoli - this is the current status of my webui git folder:

6bd6154 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #2067 from victorca25/esrgan_mod

I placed the SD 1.5 Inpainting checkpoint in the /models/stable-diffusion/ folder, and selected it from the dropdown. I'm getting this error:

  File ".\venv\lib\site-packages\torch\nn\modules\module.py", line 1604, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LatentDiffusion:
        size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 9, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).

[anwac]

Same here. To reproduce you have to have the term 'inpainting' in the file name of the ckpt. (eg SD1.5-inpainting.ckpt)

The same model file will load under a different name, for instance 'v1-5-pruned-emaonly.ckpt'. My guess is that it then simply does not have the inpainting functionality that it is been praised for.

Linux ubu3 5.15.0-52-generic #58-Ubuntu SMP Thu Oct 13 08:03:55 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

NVIDIA-SMI 510.85.02 Driver Version: 510.85.02 CUDA Version: 11.6

[pdizzl]

Tried a fresh clone and also getting this error still.

E: I'd accidentally renamed the 1.5 inpainting model. For anyone still looking for a fix to this I renamed it back to "sd-v1-5-inpainting.ckpt" and it's working again.

[anwac]

Renaming v1-5-pruned-emaonly.ckpt (size 4265380512, shae256: cc6cb27103417325ff94f52b7a5d2dde45a7515b25c255d8e396c90014281516 ) to *inpainting.ckpt was the root cause for me.

After I downloaded https://huggingface.co/runwayml/stable-diffusion-inpainting/blob/main/sd-v1-5-inpainting.ckpt (size: 4265437280, sha256: c6bbc15e3224e6973459ba78de4998b80b50112b0ae5b5c67113d56b4e366b19) the error did go away. Can't speak for the functionality yet.

[zer0TF]

Found the culprit - your inpainting .ckpt file needs to end with inpainting.ckpt:

https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/9553a7e071351455074bac255c529e1184c5ee24/modules/sd_hijack_inpainting.py#L320

Still a bug, since this is a hack and not documented.