[Bug]: Can't merge runwayml inpainting 1.5 model with any standard model

[shadinx2]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

I tried to merge a custom model with the inpainting model provided by runwayml. I also tried merging stable diffusion v1.5 with the inpainting model. Neither worked.

Steps to reproduce the problem

1. Go to merge tab.
2. Select any normal model and the inpainting model then press merge(I used weighted sum with 0.5 multiplier and tried with both save as float16 and without).
3. An error should appear.

What should have happened?

A new inapinting model should have appeared.

Commit where the problem happens

1ef32c8b8fa3e16a1e7b287eb19d4fc943d1f2a5

What platforms do you use to access UI ?

Windows

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--medvram --opt-split-attention --deepdanbooru --xformers

Additional information, context and logs

Error loading/saving model file: Traceback (most recent call last): File "C:\Users\Dimitrie\stable-diffusion-webui\modules\ui.py", line 1712, in modelmerger results = modules.extras.run_modelmerger(*args) File "C:\Users\Dimitrie\stable-diffusion-webui\modules\extras.py", line 236, in run_modelmerger theta_0[key] = theta_func2(theta_0[key], theta_1[key], multiplier) File "C:\Users\Dimitrie\stable-diffusion-webui\modules\extras.py", line 187, in weighted_sum return ((1 - alpha) * theta0) + (alpha * theta1) RuntimeError: The size of tensor a (4) must match the size of tensor b (9) at non-singleton dimension 1

Traceback (most recent call last): File "C:\Users\Dimitrie\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 275, in run_predict output = await app.blocks.process_api( File "C:\Users\Dimitrie\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 791, in process_api predictions = self.postprocess_data(fn_index, result["prediction"], state) File "C:\Users\Dimitrie\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 742, in postprocess_data if predictions[i] is components._Keywords.FINISHED_ITERATING: IndexError: list index out of range

[kiron111]

maybe you can try "git switch master" to switch back to the master branch I just made it work to merge to the main branch

[GaggiX]

The runway model is different than the standard model, its unet takes 9 latent channels instead of 4, so there is no way to merge them, their architecture is just different.

[ntrpnr]

If we merge on 4 channels and keep 5 intact, it wouldn't work?

[mayofiddler]

It needs the extra inputs

For inpainting, the UNet has 5 additional input channels (4 for the encoded masked-image and 1 for the mask itself)

[shadinx2]

Can't the algorithm be modified to copy the result from the input channels from the inpaint model if it can't find them on the other model? Maybe an extra check to select below or in the settings for such cases.

[ntrpnr]

Well it is certainly possible to merge, but I haven't gotten any good results yet. Feel free to try it:

https://gist.github.com/ntrpnr/f5cc57090d5be82284b5a72db7807136

Edit: Accidentally published the wrong version. Fixed now. Results are promising.

[ntrpnr]

Hey @AUTOMATIC1111, just want to make you aware that we have a solution here

[shadinx2]

The speed at which everyone works is astonishing.

[ntrpnr]

Opened PR: https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/3788

[mezotaken]

Now supported when inpainting model is "A".