Add support for merging inpainting models

[ntrpnr]

[mayofiddler]

Curious as to the final output of this. Will it be a unet with the same inputs as the model you are merging with? So that embedding and so on work with it as if it is one of the "standard" models? Or will it have the extra inputs of the inpainting model?

[ntrpnr]

It will have the extra inputs of the inpainting model as before the merge, since those are needed for inpainting. Other inputs are merged based on the multiplier.

[mayofiddler]

OK thanks @ntrpnr :)

[shadinx2]

I tested it out and the merger works. Also the new inpainting model produced seems to make good results as per my limited testing so far.

[AUTOMATIC1111]

Should this not be possible to do without checkbos and without the big if/else block? Just checking the sizes of certain layer dicts to determine if you have an inpainting model or not.

[ntrpnr]

@AUTOMATIC1111 Yes it is possible without checkbox. The reason for doing it this way:

• The user will be aware that it’s now possible to merge inpainting models.
• This approach leaves the current merge logic untouched and will change fewer lines of code.

[random-thoughtss]

One thing you can do to avoid adding another if-else layer is to just pad any weights that are not the 9-input weights before hand. Then you can re-use the assignment code without modification.

FIRST_WEIGHT = "model.diffusion_model.input_blocks.0.0.weight"
if theta_0[FIRST_WEIGHT].shape[1] == 4:
    theta_0[FIRST_WEIGHT] = torch.nn.functional.pad(theta_0[FIRST_WEIGHT], (0, 0, 0, 0, 0, 5))

[ntrpnr]

@random-thoughtss Thanks, but I’m not sure this improves readability. At least not to me as a Python novice.

But it might be a cleaner approach.

[snsd21116890]

I had try it and sucessful merger my model with inpainting models.But it failed when I loaded it.

Traceback (most recent call last): File "K:\SDUI3\venv\lib\site-packages\gradio\routes.py", line 283, in run_predict output = await app.blocks.process_api( File "K:\SDUI3\venv\lib\site-packages\gradio\blocks.py", line 936, in process_api result = await self.call_function(fn_index, inputs, iterator) File "K:\SDUI3\venv\lib\site-packages\gradio\blocks.py", line 777, in call_function prediction = await anyio.to_thread.run_sync( File "K:\SDUI3\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "K:\SDUI3\venv\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "K:\SDUI3\venv\lib\site-packages\anyio_backends_asyncio.py", line 867, in run result = context.run(func, *args) File "K:\SDUI3\modules\ui.py", line 1633, in fn=lambda value, k=k: run_settings_single(value, key=k), File "K:\SDUI3\modules\ui.py", line 1474, in run_settings_single opts.data_labels[key].onchange() File "K:\SDUI3\webui.py", line 42, in f res = func(*args, **kwargs) File "K:\SDUI3\webui.py", line 83, in shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights())) File "K:\SDUI3\modules\sd_models.py", line 288, in reload_model_weights load_model_weights(sd_model, checkpoint_info) File "K:\SDUI3\modules\sd_models.py", line 181, in load_model_weights model.load_state_dict(sd, strict=False) File "K:\SDUI3\venv\lib\site-packages\torch\nn\modules\module.py", line 1604, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for LatentDiffusion: size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 9, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).

[ntrpnr]

@snsd21116890 Did you name it [...]inpainting.ckpt?

[snsd21116890]

@snsd21116890 Did you name it [...]inpainting.ckpt?

It works now,thank you.It seems that the merger only enable in Weighted sum method.I had a error when I use Add difference method.Is possible the merger with inpainting model in Add difference method?

[ilazarte]

Any word on this merge? Seems like a very high value feature.

[ntrpnr]

Currently, I don't have the time to continue working on this. If anyone else wants to go for it, please go ahead.

[ilazarte]

Thank you Automatic!