comfyanonymous
Why can't ComfyUI load SD3-Controlnet-Inpainting?
Your question
Error when loading SD3-Controlnet-Inpainting in ComfyUI (default Load ControlNet Model Node):
sd3 controlnet inpaint Error(s) in loading state_dict for ControlNet: size mismatch for pos_embed_input.proj.weight: copying a param with shape torch.Size([1536, 17, 2, 2]) from checkpoint, the shape in current model is torch.Size([1536, 16, 2, 2]).
diffusers: 0.30.2
Logs
[2024-09-11 22:53] got prompt
[2024-09-11 22:55] !!! Exception during processing !!! Error(s) in loading state_dict for ControlNet:
size mismatch for pos_embed_input.proj.weight: copying a param with shape torch.Size([1536, 17, 2, 2]) from checkpoint, the shape in current model is torch.Size([1536, 16, 2, 2]).
[2024-09-11 22:55] Traceback (most recent call last):
File "E:\Program Files\ComfyUI_windows_portable\ComfyUI\execution.py", line 317, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Program Files\ComfyUI_windows_portable\ComfyUI\execution.py", line 192, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Program Files\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)
File "E:\Program Files\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Program Files\ComfyUI_windows_portable\ComfyUI\nodes.py", line 758, in load_controlnet
controlnet = comfy.controlnet.load_controlnet(controlnet_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Program Files\ComfyUI_windows_portable\ComfyUI\comfy\controlnet.py", line 529, in load_controlnet
return load_controlnet_mmdit(controlnet_data) #SD3 diffusers controlnet
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Program Files\ComfyUI_windows_portable\ComfyUI\comfy\controlnet.py", line 414, in load_controlnet_mmdit
control_model = controlnet_load_state_dict(control_model, new_sd)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Program Files\ComfyUI_windows_portable\ComfyUI\comfy\controlnet.py", line 397, in controlnet_load_state_dict
missing, unexpected = control_model.load_state_dict(sd, strict=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Program Files\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 2189, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ControlNet:
size mismatch for pos_embed_input.proj.weight: copying a param with shape torch.Size([1536, 17, 2, 2]) from checkpoint, the shape in current model is torch.Size([1536, 16, 2, 2]).
Other
All other SD3 Controlnet models including Alimama's SD3-Controlnet-Softedge work fine.
In new diffusers for official, I find a new parameter "extra_conditioning_channels" for SD3ControlNetModel.
That may be a new feature in SD3ControlNetModel which comfyUI not supported.
Alimama SD3 inpating controlnet had config with "extra_conditioning_channels=1"
official diffusers
I would try to create a PR later.
Great. Maybe it's the same issue for Alimama's SD3 and Flux's ControlNet Inpaint models.
I have fixed the parameter passing problem of pos_embed_input.proj.weight.
But there are more problems here,
The input of Alibaba's SD3 ControlNet inpaint model expands the input latent channel😂, so the input channel of the ControlNet inpaint model is expanded to 17😂😂😂😂😂, and this expanded channel is actually the mask of the inpaint target. The comfyUI process needs to be modified to pass this mask to the latent input in ControlNet.I think it would need to extend a custom_model to handle this.
alimama has SD3 and Flux controlnet inpainting models, but obviously they don't use ComfyUI, so their models can't run on ComfyUI.
The community is hoping for xinsir's controlnet-union for SD3/Flux https://huggingface.co/xinsir/controlnet-union-sdxl-1.0
Yeah, I like Xinsir. But why Alimama's Alimama's SD3-Controlnet-Softedge works fine, Is it just a coincidence? I've created a discussion with them but no response.
Yeah, I like Xinsir. But why Alimama's Alimama's SD3-Controlnet-Softedge works fine, Is it just a coincidence? I've created a discussion with them but no response.
That's because the layers and inputs of SD3-controlnet-Softedge are of standard size, but the inpaint model is not. I also learned about extra_conditioning_channels by tracing the code of this model. In order to be compatible with alimama's sd3-controlnet-inpaint model, it is best to extend a new comfyUI node.
@jakechai I wrote a demo that is compatible with the Alimama-SD3-ControlNet-Inpaint model. You can put it in custom_nodes to see it.
https://github.com/zhiselfly/ComfyUI-Alimama-ControlNet-compatible
Please use "Alimama SD3 Inpaint ControlNet Loader" to load the ControlNet model and "Alimama SD3 Inpaint ControlNet Apply Advanced" to apply the ControlNet.
@jakechai I wrote a demo that is compatible with the Alimama-SD3-ControlNet-Inpaint model. You can put it in custom_nodes to see it. https://github.com/zhiselfly/ComfyUI-Alimama-ControlNet-compatible Please use "Alimama SD3 Inpaint ControlNet Loader" to load the ControlNet model and "Alimama SD3 Inpaint ControlNet Apply Advanced" to apply the ControlNet.
Does it work with the Flux controlnet too?
@zhiselfly I can’t wait to test it out. I've opened an issue in your repository.
@jakechai I wrote a demo that is compatible with the Alimama-SD3-ControlNet-Inpaint model. You can put it in custom_nodes to see it. https://github.com/zhiselfly/ComfyUI-Alimama-ControlNet-compatible Please use "Alimama SD3 Inpaint ControlNet Loader" to load the ControlNet model and "Alimama SD3 Inpaint ControlNet Apply Advanced" to apply the ControlNet.
Does it work with the Flux controlnet too?
I don’t have a Flux model😂. I need to find some time to download and see how other libraries can be migrated into comfyUI.
https://github.com/zhiselfly/ComfyUI-Alimama-ControlNet-compatible It works for SD3. Great job! @zhiselfly
https://github.com/comfyanonymous/ComfyUI/commit/f48e390032f8d27a450a35ef4aa4b775b078cbf9
