[Bug]: mat1 and mat2 shapes cannot be multiplied

[VeganMooseMeat]

Is there an existing issue for this?

• [x] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

I am getting an error when trying to run openpose with v2-1_768-ema-pruned.safetensors. It only happens with this SD checkpoint. The error is "RuntimeError: mat1 and mat2 shapes cannot be multiplied (77x1024 and 768x320)"

This error has been reported and closed before by others, but the pulling did not fix the issue.

Steps to reproduce the problem

Text2img settings: Sampling mthd = Euler a sampling steps = 50 no boxes checked width = 1920 height = 1080 CFG scale = 7

Control model settings: model = openpose, preprocessor = none. Weight 1, Guidance start = 0, end = 1 resize mode = just resize, this happens on all other modes. Canvas width = 1024, height = 576

What should have happened?

Pose should have been applied to prompts

Commit where the problem happens

webui: https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/a9fed7c364061ae6efb37f797b6b522cb3cf7aa2 controlnet: https://github.com/Mikubill/sd-webui-controlnet/commit/de8fdeff373695431440b9c29e9a6fe24155e795

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--xformers

Console logs

venv "C:\Users\dalto\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Commit hash: a9fed7c364061ae6efb37f797b6b522cb3cf7aa2
Installing requirements for Web UI


Initializing Dreambooth
If submitting an issue on github, please provide the below text for debugging purposes:

Python revision: 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Dreambooth revision: 1b8437cb7e3f2b963058caed0abf237a94add0c7
SD-WebUI revision: a9fed7c364061ae6efb37f797b6b522cb3cf7aa2

Successfully installed protobuf-3.19.6
Successfully installed transformers-4.27.1

[+] torch version 1.13.1+cu117 installed.
[+] torchvision version 0.14.1+cu117 installed.
[+] xformers version 0.0.17.dev476 installed.
[+] accelerate version 0.17.1 installed.
[+] diffusers version 0.14.0 installed.
[+] transformers version 4.27.1 installed.
[+] bitsandbytes version 0.35.4 installed.


Launching Web UI with arguments: --xformers
Loading weights [dcd690123c] from C:\Users\dalto\stable-diffusion-webui\models\Stable-diffusion\v2-1_768-ema-pruned.safetensors
Creating model from config: C:\Users\dalto\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\configs\stable-diffusion\v2-inference-v.yaml
LatentDiffusion: Running in v-prediction mode
DiffusionWrapper has 865.91 M params.
Applying xformers cross attention optimization.
Textual inversion embeddings loaded(0):
Model loaded in 10.9s (load weights from disk: 0.3s, find config: 3.1s, create model: 0.3s, apply weights to model: 3.5s, apply half(): 1.3s, move model to device: 1.0s, load textual inversion embeddings: 1.4s).
[VRAMEstimator] Loaded benchmark data.
CUDA SETUP: Loading binary C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cudaall.dll...
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 33.1s (import gradio: 1.5s, import ldm: 0.9s, other imports: 1.7s, list extensions: 0.5s, setup codeformer: 0.1s, load scripts: 5.4s, load SD checkpoint: 11.1s, create ui: 11.8s).
Traceback (most recent call last):
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 337, in run_predict
    output = await app.get_blocks().process_api(
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1015, in process_api
    result = await self.call_function(
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 833, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "C:\Users\dalto\stable-diffusion-webui\extensions\a1111-stable-diffusion-webui-vram-estimator\scripts\vram_estimator.py", line 342, in estimate_vram_txt2img
    vram_estimate = curves["txt2img"].estimate(final_width * final_height, batch_size)
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
Loading model: control_openpose-fp16 [9ca67cc5]
Loaded state_dict from [C:\Users\dalto\stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_openpose-fp16.safetensors]
ControlNet model control_openpose-fp16 [9ca67cc5] loaded.
Loading preprocessor: none
  0%|                                                                                           | 0/50 [00:02<?, ?it/s]
Error completing request
Arguments: ('task(9yemejwv5es4s74)', 'colorful, saturated, (ultra detailed face and eyes:1.1), (album art:1.1), nostalgia, (cinematic:1.5), moody, dramatic lighting, (photo:0.6), majestic, oil painting, high detail, soft focus, (neon:0.7), golden hour, bokeh, (centered:1.5), (rough brushstrokes:1.3), (concept art:1.5), rimlight', 'cartoon, 3d, zombie, disfigured, deformed, extra limbs, b&w, black and white, duplicate, morbid, mutilated, cropped, out of frame, extra fingers, mutated hands, mutation, extra limbs, clone, out of frame, too many fingers, long neck, tripod, photoshop, video game, tiling, cut off head, patterns, borders, (frame:1.4), symmetry, intricate, signature, text, watermark', [], 50, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 1080, 1920, False, 0.7, 2, 'Latent', 0, 0, 0, [], 0, '\n    <div style="padding: 10px">\n      <div>Estimated VRAM usage: <span style="color: rgb(255.00, 0.00, 204.00)">12360.90 MB / 10240 MB (120.71%)</span></div>\n      <div>(4159 MB system + 7456.28 MB used)</div>\n    </div>\n    ', <scripts.external_code.ControlNetUnit object at 0x000001FFE9B7E230>, <scripts.external_code.ControlNetUnit object at 0x000001FFE9BD6AA0>, <scripts.external_code.ControlNetUnit object at 0x000001FFE9BD5270>, <scripts.external_code.ControlNetUnit object at 0x000001FFE9BF4BB0>, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0, None, False, None, False, None, False, None, False, 50) {}
Traceback (most recent call last):
  File "C:\Users\dalto\stable-diffusion-webui\modules\call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "C:\Users\dalto\stable-diffusion-webui\modules\call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\modules\txt2img.py", line 56, in txt2img
    processed = process_images(p)
  File "C:\Users\dalto\stable-diffusion-webui\modules\processing.py", line 486, in process_images
    res = process_images_inner(p)
  File "C:\Users\dalto\stable-diffusion-webui\modules\processing.py", line 636, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
  File "C:\Users\dalto\stable-diffusion-webui\modules\processing.py", line 836, in sample
    samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
  File "C:\Users\dalto\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 351, in sample
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "C:\Users\dalto\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 227, in launch_sampling
    return func()
  File "C:\Users\dalto\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 351, in <lambda>
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 138, in forward
    x_out[a:b] = self.inner_model(x_in[a:b], sigma_in[a:b], cond={"c_crossattn": c_crossattn, "c_concat": [image_cond_in[a:b]]})
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 167, in forward
    return self.get_v(input * c_in, self.sigma_to_t(sigma), **kwargs) * c_out + input * c_skip
  File "C:\Users\dalto\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 177, in get_v
    return self.inner_model.apply_model(x, t, cond)
  File "C:\Users\dalto\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
    setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
  File "C:\Users\dalto\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
    return self.__orig_func(*args, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
    x_recon = self.model(x_noisy, t, **cond)
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1329, in forward
    out = self.diffusion_model(x, t, context=cc)
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 233, in forward2
    return forward(*args, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\hook.py", line 176, in forward
    control = param.control_model(x=x_in, hint=param.hint_cond, timesteps=timesteps, context=context)
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 115, in forward
    return self.control_model(*args, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 383, in forward
    h = module(h, emb, context)
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward
    x = layer(x, context)
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 324, in forward
    x = block(x, context=context[i])
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 259, in forward
    return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
  File "C:\Users\dalto\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 114, in checkpoint
    return CheckpointFunction.apply(func, len(inputs), *args)
  File "C:\Users\dalto\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 129, in forward
    output_tensors = ctx.run_function(*ctx.input_tensors)
  File "C:\Users\dalto\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 263, in _forward
    x = self.attn2(self.norm2(x), context=context) + x
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\modules\sd_hijack_optimizations.py", line 332, in xformers_attention_forward
    k_in = self.to_k(context_k)
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\dalto\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 197, in lora_Linear_forward
    return lora_forward(self, input, torch.nn.Linear_forward_before_lora(self, input))
  File "C:\Users\dalto\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (77x1024 and 768x320)

Additional information

No response

[tsugg]

pretty sure ControlNet only works for SD15

[AIAMIAUTHOR]

controlnet works with sd 2.1 but some loras are not comaptible yet, control_openpose-fp16 is sd 1.5, try to use openpose-sd21-safe.safetensors from thibaud, if using wd models, use wd15beta(sd 2.1) controlnet from furusu

[VeganMooseMeat]

controlnet works with sd 2.1 but some loras are not comaptible yet, control_openpose-fp16 is sd 1.5, try to use openpose-sd21-safe.safetensors from thibaud, if using wd models, use wd15beta(sd 2.1) controlnet from furusu

[captainhalf]

pretty sure ControlNet only works for SD15

this one comment just cleared all the hurdles ..and i ended up geting myself a batch of img2img video ..my very first. thanks a lot ....

[vaagi]

pretty sure ControlNet only works for SD15

this one comment just cleared all the hurdles ..and i ended up geting myself a batch of img2img video ..my very first. thanks a lot ....

ive recently installed stable diffusion and controlnet i am an architect trying to learn AI but while using freedomredmon and controlnet using 2 controlnet one to inpaint for the base image and then the other scribble so that the scribbled sketch can be used by AI for my inpaint base image using this it gives me this error please help