RuntimeError: mat1 and mat2 shapes cannot be multiplied (616x1024 and 768x320)

[Rartist]

Thanks for this, I hope I can get it to work soon!

I tried the Scribble Model, but I got this error once I ran it: RuntimeError: mat1 and mat2 shapes cannot be multiplied (616x1024 and 768x320)

I resized the input image to 512 x 704 pixels and set the normal Width & Height accordingly. I tried running it without any Preprocessor or with the Fake_Scribble processor, but got the same error message both times. Weight left at 1, Scribble Mode On & Off tried.

[Mikubill]

Could you share the full output when this error occurred, or the input args of process func? Seems like some configure not compatible with controlnet

[Rartist]

Might have been because I had a V2-1 Model loaded. With a 1-5 Model I don't get an error anymore, but I don't see the image change (at all), either. It generates 1 normal image and gives me my input image back (unchanged).

[Xenocyde3000]

I get a similar RuntimeError: mat1 and mat2 shapes cannot be multiplied (77x1024 and 768x320)

Tried to crop the input image to match the aspect ratio of the rendered image (768 x 512), too. I'm using an SD 2.1 model, though. Could this be the problem? Or maybe I need a 1:1 aspect ratio?

[YooPita]

I also ran into an error using 2.1. After switching to 1.5 everything began to work.

[ll2pakll]

I have the same problem. But what is interesting, initially, immediately after installing the extension, everything worked on the model 2.1. But after restarting it, I started getting this error. So the problem is not the model, but something else that changes with the model change. Just in case, here's my conclusion. Maybe someone can still solve the problem? It is obvious that the program is trying to multiply tensors that do not match in dimensions, but how to fix it, I do not have enough knowledge

Model loaded in 10.1s (find config: 2.1s, load config: 0.2s, create model: 0.6s, apply weights to model: 1.3s, apply half(): 4.8s, move model to device: 1.0s).
Loading model: control_sd15_canny [fef5e48e]
Loaded state_dict from [F:\Work area\stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_sd15_canny.pth]
ControlNet model control_sd15_canny [fef5e48e] loaded.
Loading preprocessor: canny
  0%|                                                                                           | 0/16 [00:00<?, ?it/s]
Error completing request
Arguments: ('task(finyozvsy56tlzj)', 0, 'Girl, cyberpunk, 8k, neon', '', [], <PIL.Image.Image image mode=RGBA size=512x512 at 0x2248D7FB850>, None, None, None, None, None, None, 20, 0, 4, 0, 1, False, False, 1, 1, 7, 1.5, 0.75, -1.0, -1.0, 0, 0, 0, False, 512, 512, 0, 0, 32, 0, '', '', '', [], 0, True, 'canny', 'control_sd15_canny [fef5e48e]', 1, {'image': array([[[159, 213, 242],
        [153, 213, 250],
        [146, 213, 248],
        ...,
        [182, 230, 247],
        [184, 233, 252],
        [191, 229, 249]],

       [[158, 216, 249],
        [153, 216, 253],
        [147, 210, 245],
        ...,
        [183, 228, 247],
        [184, 231, 251],
        [187, 232, 253]],

       [[155, 213, 247],
        [155, 216, 250],
        [150, 211, 246],
        ...,
        [184, 229, 250],
        [185, 229, 250],
        [185, 227, 246]],

       ...,

       [[ 74,  38,  41],
        [ 62,  38,  48],
        [ 56,  41,  52],
        ...,
        [186, 124, 113],
        [181, 123, 114],
        [174, 118, 112]],

       [[ 71,  36,  37],
        [ 60,  35,  41],
        [ 53,  40,  48],
        ...,
        [178, 109,  96],
        [180, 114, 103],
        [173, 112, 103]],

       [[ 66,  36,  37],
        [ 59,  32,  39],
        [ 53,  36,  49],
        ...,
        [171, 101,  88],
        [173, 106,  95],
        [162, 108,  98]]], dtype=uint8), 'mask': array([[[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       ...,

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]],

       [[  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        ...,
        [  0,   0,   0, 255],
        [  0,   0,   0, 255],
        [  0,   0,   0, 255]]], dtype=uint8)}, False, 'Just Resize', False, True, 512, 100, 200, 1, False, '<ul>\n<li><code>CFG Scale</code> should be 2 or lower.</li>\n</ul>\n', True, True, '', '', True, 50, True, 1, 0, False, 4, 1, 'None', '<p style="margin-bottom:0.75em">Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8</p>', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, '', '<p style="margin-bottom:0.75em">Will upscale the image by the selected scale factor; use width and height sliders to set tile size</p>', 64, 0, 2, 1, '', 0, '', 0, '', True, False, False, False, 0) {}
Traceback (most recent call last):
  File "F:\Work area\stable-diffusion-webui\modules\call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "F:\Work area\stable-diffusion-webui\modules\call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "F:\Work area\stable-diffusion-webui\modules\img2img.py", line 171, in img2img
    processed = process_images(p)
  File "F:\Work area\stable-diffusion-webui\modules\processing.py", line 486, in process_images
    res = process_images_inner(p)
  File "F:\Work area\stable-diffusion-webui\modules\processing.py", line 632, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
  File "F:\Work area\stable-diffusion-webui\modules\processing.py", line 1048, in sample
    samples = self.sampler.sample_img2img(self, self.init_latent, x, conditioning, unconditional_conditioning, image_conditioning=self.image_conditioning)
  File "F:\Work area\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 322, in sample_img2img
    samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
  File "F:\Work area\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 225, in launch_sampling
    return func()
  File "F:\Work area\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 322, in <lambda>
    samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "F:\Work area\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\Work area\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 117, in forward
    x_out = self.inner_model(x_in, sigma_in, cond={"c_crossattn": [cond_in], "c_concat": [image_cond_in]})
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\Work area\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 167, in forward
    return self.get_v(input * c_in, self.sigma_to_t(sigma), **kwargs) * c_out + input * c_skip
  File "F:\Work area\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 177, in get_v
    return self.inner_model.apply_model(x, t, cond)
  File "F:\Work area\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
    setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
  File "F:\Work area\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
    return self.__orig_func(*args, **kwargs)
  File "F:\Work area\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
    x_recon = self.model(x_noisy, t, **cond)
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\Work area\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1329, in forward
    out = self.diffusion_model(x, t, context=cc)
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\Work area\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 190, in forward2
    return forward(*args, **kwargs)
  File "F:\Work area\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 133, in forward
    control = param.control_model(x=x_in, hint=param.hint_cond, timesteps=timesteps, context=context)
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\Work area\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 115, in forward
    return self.control_model(*args, **kwargs)
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\Work area\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 383, in forward
    h = module(h, emb, context)
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\Work area\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward
    x = layer(x, context)
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\Work area\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 324, in forward
    x = block(x, context=context[i])
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\Work area\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 259, in forward
    return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
  File "F:\Work area\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 114, in checkpoint
    return CheckpointFunction.apply(func, len(inputs), *args)
  File "F:\Work area\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 129, in forward
    output_tensors = ctx.run_function(*ctx.input_tensors)
  File "F:\Work area\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 263, in _forward
    x = self.attn2(self.norm2(x), context=context) + x
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\Work area\stable-diffusion-webui\modules\sd_hijack_optimizations.py", line 332, in xformers_attention_forward
    k_in = self.to_k(context_k)
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "F:\Work area\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 178, in lora_Linear_forward
    return lora_forward(self, input, torch.nn.Linear_forward_before_lora(self, input))
  File "f:\Work area\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (154x1024 and 768x320)

[jaycecd]

@Mikubill The same error message reporting! Please tell me why this issue was turned off? I've pulled the latest code but it still doesn't work with SD2.1(v2-1_512-ema-pruned.ckpt [88ecb78256])

[deloyed]

Anyone could fix the problem?

[ljleb]

It seems this was caused by the active model and enabled controlnet units not being the same type of model. For example, mixing v2.1 with v1.5 models would result into this error, according to prior discussion IIUC.

[deloyed]

how do i know the units for different models

[ljleb]

It's in the model name. models for v1.5 have sd15 in their name.

I have the same problem. But what is interesting, initially, immediately after installing the extension, everything worked on the model 2.1. But after restarting it, I started getting this error. So the problem is not the model, but something else that changes with the model change.

Maybe this isn't the issue though, according to this ^. It could still be related however.

[korronald]

I have a Very similar problem, with all models.

RuntimeError: linear(): input and weight.T shapes cannot be multiplied (77x1024 and 768x320)

Mac M1, Stable Diffusion 2.1, Automatic 1111, When changing webui-user.sh acording to others suggestion (adding --no-half etc.) I'm ending up with problem:

RuntimeError: Placeholder storage has not been allocated on MPS device!

[korronald]

looks like it works after changing models to 21 models. Thanmks!