Model merge broken | clip encode errors | Multi lora errors | much more

[kakachiex2]

ComfyUI is completely broken, All my workflow with model merge multi lora, average conditioning is giving lots of errors, and warning, Noised image k-sampler generation. In this image the first generation was done correctly after this first one all the other give this noised, distorted image, and I get a lot of warning that don't show before.

First generation image and the generating noised image.

Errors and warning: WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.1.transformer_blocks.0.ff.net.2.weight WEIGHT NOT MERGED torch.Size([640, 2560]) != torch.Size([640]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.1.transformer_blocks.0.ff.net.2.weight WEIGHT NOT MERGED torch.Size([640, 2560]) != torch.Size([640]) ERROR lora diffusion_model.output_blocks.5.1.transformer_blocks.0.ff.net.2.weight shape '[640]' is invalid for input of size 1638400 WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.1.transformer_blocks.1.attn1.to_out.0.weight WEIGHT NOT MERGED torch.Size([640, 640]) != torch.Size([640]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.1.transformer_blocks.1.attn1.to_out.0.weight WEIGHT NOT MERGED torch.Size([640, 640]) != torch.Size([640]) ERROR lora diffusion_model.output_blocks.5.1.transformer_blocks.1.attn1.to_out.0.weight shape '[640]' is invalid for input of size 409600 WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.1.transformer_blocks.1.attn2.to_out.0.weight WEIGHT NOT MERGED torch.Size([640, 640]) != torch.Size([640]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.1.transformer_blocks.1.attn2.to_out.0.weight WEIGHT NOT MERGED torch.Size([640, 640]) != torch.Size([640]) ERROR lora diffusion_model.output_blocks.5.1.transformer_blocks.1.attn2.to_out.0.weight shape '[640]' is invalid for input of size 409600 WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.1.transformer_blocks.1.ff.net.0.proj.weight WEIGHT NOT MERGED torch.Size([5120, 640]) != torch.Size([5120]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.1.transformer_blocks.1.ff.net.0.proj.weight WEIGHT NOT MERGED torch.Size([5120, 640]) != torch.Size([5120]) ERROR lora diffusion_model.output_blocks.5.1.transformer_blocks.1.ff.net.0.proj.weight shape '[5120]' is invalid for input of size 3276800 WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.1.transformer_blocks.1.ff.net.2.weight WEIGHT NOT MERGED torch.Size([640, 2560]) != torch.Size([640]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.1.transformer_blocks.1.ff.net.2.weight WEIGHT NOT MERGED torch.Size([640, 2560]) != torch.Size([640]) ERROR lora diffusion_model.output_blocks.5.1.transformer_blocks.1.ff.net.2.weight shape '[640]' is invalid for input of size 1638400 WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.1.proj_out.weight WEIGHT NOT MERGED torch.Size([640, 640]) != torch.Size([640]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.1.proj_out.weight WEIGHT NOT MERGED torch.Size([640, 640]) != torch.Size([640]) ERROR lora diffusion_model.output_blocks.5.1.proj_out.weight shape '[640]' is invalid for input of size 409600 WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.2.conv.weight WEIGHT NOT MERGED torch.Size([640, 640, 3, 3]) != torch.Size([640]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.5.2.conv.weight WEIGHT NOT MERGED torch.Size([640, 640, 3, 3]) != torch.Size([640]) ERROR lora diffusion_model.output_blocks.5.2.conv.weight shape '[640]' is invalid for input of size 3686400 WARNING SHAPE MISMATCH diffusion_model.output_blocks.6.0.in_layers.2.weight WEIGHT NOT MERGED torch.Size([320, 960, 3, 3]) != torch.Size([320]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.6.0.in_layers.2.weight WEIGHT NOT MERGED torch.Size([320, 960, 3, 3]) != torch.Size([320]) ERROR lora diffusion_model.output_blocks.6.0.in_layers.2.weight shape '[320]' is invalid for input of size 2764800 WARNING SHAPE MISMATCH diffusion_model.output_blocks.6.0.emb_layers.1.weight WEIGHT NOT MERGED torch.Size([320, 1280]) != torch.Size([320]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.6.0.emb_layers.1.weight WEIGHT NOT MERGED torch.Size([320, 1280]) != torch.Size([320]) ERROR lora diffusion_model.output_blocks.6.0.emb_layers.1.weight shape '[320]' is invalid for input of size 409600 WARNING SHAPE MISMATCH diffusion_model.output_blocks.6.0.out_layers.3.weight WEIGHT NOT MERGED torch.Size([320, 320, 3, 3]) != torch.Size([320]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.6.0.out_layers.3.weight WEIGHT NOT MERGED torch.Size([320, 320, 3, 3]) != torch.Size([320]) ERROR lora diffusion_model.output_blocks.6.0.out_layers.3.weight shape '[320]' is invalid for input of size 921600 WARNING SHAPE MISMATCH diffusion_model.output_blocks.6.0.skip_connection.weight WEIGHT NOT MERGED torch.Size([320, 960, 1, 1]) != torch.Size([320]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.6.0.skip_connection.weight WEIGHT NOT MERGED torch.Size([320, 960, 1, 1]) != torch.Size([320]) ERROR lora diffusion_model.output_blocks.6.0.skip_connection.weight shape '[320]' is invalid for input of size 307200 WARNING SHAPE MISMATCH diffusion_model.output_blocks.7.0.in_layers.2.weight WEIGHT NOT MERGED torch.Size([320, 640, 3, 3]) != torch.Size([320]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.7.0.in_layers.2.weight WEIGHT NOT MERGED torch.Size([320, 640, 3, 3]) != torch.Size([320]) ERROR lora diffusion_model.output_blocks.7.0.in_layers.2.weight shape '[320]' is invalid for input of size 1843200 WARNING SHAPE MISMATCH diffusion_model.output_blocks.7.0.emb_layers.1.weight WEIGHT NOT MERGED torch.Size([320, 1280]) != torch.Size([320]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.7.0.emb_layers.1.weight WEIGHT NOT MERGED torch.Size([320, 1280]) != torch.Size([320]) ERROR lora diffusion_model.output_blocks.7.0.emb_layers.1.weight shape '[320]' is invalid for input of size 409600 WARNING SHAPE MISMATCH diffusion_model.output_blocks.7.0.out_layers.3.weight WEIGHT NOT MERGED torch.Size([320, 320, 3, 3]) != torch.Size([320]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.7.0.out_layers.3.weight WEIGHT NOT MERGED torch.Size([320, 320, 3, 3]) != torch.Size([320]) ERROR lora diffusion_model.output_blocks.7.0.out_layers.3.weight shape '[320]' is invalid for input of size 921600 WARNING SHAPE MISMATCH diffusion_model.output_blocks.7.0.skip_connection.weight WEIGHT NOT MERGED torch.Size([320, 640, 1, 1]) != torch.Size([320]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.7.0.skip_connection.weight WEIGHT NOT MERGED torch.Size([320, 640, 1, 1]) != torch.Size([320]) ERROR lora diffusion_model.output_blocks.7.0.skip_connection.weight shape '[320]' is invalid for input of size 204800 WARNING SHAPE MISMATCH diffusion_model.output_blocks.8.0.in_layers.2.weight WEIGHT NOT MERGED torch.Size([320, 640, 3, 3]) != torch.Size([320]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.8.0.in_layers.2.weight WEIGHT NOT MERGED torch.Size([320, 640, 3, 3]) != torch.Size([320]) ERROR lora diffusion_model.output_blocks.8.0.in_layers.2.weight shape '[320]' is invalid for input of size 1843200 WARNING SHAPE MISMATCH diffusion_model.output_blocks.8.0.emb_layers.1.weight WEIGHT NOT MERGED torch.Size([320, 1280]) != torch.Size([320]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.8.0.emb_layers.1.weight WEIGHT NOT MERGED torch.Size([320, 1280]) != torch.Size([320]) ERROR lora diffusion_model.output_blocks.8.0.emb_layers.1.weight shape '[320]' is invalid for input of size 409600 WARNING SHAPE MISMATCH diffusion_model.output_blocks.8.0.out_layers.3.weight WEIGHT NOT MERGED torch.Size([320, 320, 3, 3]) != torch.Size([320]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.8.0.out_layers.3.weight WEIGHT NOT MERGED torch.Size([320, 320, 3, 3]) != torch.Size([320]) ERROR lora diffusion_model.output_blocks.8.0.out_layers.3.weight shape '[320]' is invalid for input of size 921600 WARNING SHAPE MISMATCH diffusion_model.output_blocks.8.0.skip_connection.weight WEIGHT NOT MERGED torch.Size([320, 640, 1, 1]) != torch.Size([320]) WARNING SHAPE MISMATCH diffusion_model.output_blocks.8.0.skip_connection.weight WEIGHT NOT MERGED torch.Size([320, 640, 1, 1]) != torch.Size([320]) ERROR lora diffusion_model.output_blocks.8.0.skip_connection.weight shape '[320]' is invalid for input of size 204800 WARNING SHAPE MISMATCH diffusion_model.out.2.weight WEIGHT NOT MERGED torch.Size([4, 320, 3, 3]) != torch.Size([4]) WARNING SHAPE MISMATCH diffusion_model.out.2.weight WEIGHT NOT MERGED torch.Size([4, 320, 3, 3]) != torch.Size([4]) 38%|███████████████████████████████▌ | 3/8 [00:52<01:28, 17.66s/it] Processing interrupted Prompt executed in 62.01 seconds

[comfyanonymous]

Do these workflows work? https://comfyanonymous.github.io/ComfyUI_examples/model_merging/

[kakachiex2]

the screen I posted is base in those official example I get the same errors I tested all of them and others. Here is the workflow and image for testing if you have time of course:

[kakachiex2]

You can see in this video the problem

https://github.com/comfyanonymous/ComfyUI/assets/1762112/789650f6-8021-4589-ba2f-89d8951ed1ed

[kakachiex2]

The workflow is more advance "modified' vase on my others workflow.

[kakachiex2]

Here is another test > :: | Ronghua_SDXL Model | Lightning Lora :: | CR-Multi Lora Stack | SDXL | ::

• Multi Lora Stack > Bad image.
• Multi Lora Stack deactivate "Off" > Good image.

[kakachiex2]

Any news about this problem...?

[melMass]

You should post a stripped-down version of your issue, you might find the solution while doing it. I'm not familiar with sdxl lighting but I would try to bypass the lora to see if it's the issue

[kakachiex2]

I did it this workflow is the official one with some change to match my version a simplify version of my workflow to see were the problem is but do the same.

[kakachiex2]

Multi Lora not working Here is an example and a terminal log wen running the multi lora stack. WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280 WARNING: shape mismatch when trying to apply embedding, embedding will be ignored 768 != 1280

[kakachiex2]

It supposed to be better with Multi Lora but the image looks low quality and the Hight quality image is lightning model only.

• :: Low quality image > lightning lora > multi lora activated ::
• :: Hight quality image > lightning lora > multi lora deactivated ::