Doesn't work with SDXL IP-Adapter Controlnet

[ramyma]

When I use the extension with IP-adpater CN I get this error:

*** 
***        [[0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0],
***         ...,
***         [0, 0, 0],
***         [0, 0, 0],
***         [0, 0, 0]]], dtype=uint8)}, resize_mode='Crop and Resize', processor_res=0.5, threshold_a=0.5, threshold_b=0.5, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', save_detected_map=True), ControlNetUnit(input_mode=<InputMode.SIMPLE: 'simple'>, use_preview_as_input=False, batch_image_dir='', batch_mask_dir='', batch_input_gallery=[], batch_mask_gallery=[], generated_image=None, mask_image=None, hr_option='Both', enabled=False, module='None', model='None', weight=1, image=None, resize_mode='Crop and Resize', processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', save_detected_map=True), ControlNetUnit(input_mode=<InputMode.SIMPLE: 'simple'>, use_preview_as_input=False, batch_image_dir='', batch_mask_dir='', batch_input_gallery=[], batch_mask_gallery=[], generated_image=None, mask_image=None, hr_option='Both', enabled=False, module='None', model='None', weight=1, image=None, resize_mode='Crop and Resize', processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', save_detected_map=True), ControlNetUnit(input_mode=<InputMode.SIMPLE: 'simple'>, use_preview_as_input=False, batch_image_dir='', batch_mask_dir='', batch_input_gallery=[], batch_mask_gallery=[], generated_image=None, mask_image=None, hr_option='Both', enabled=False, module='None', model='None', weight=1, image=None, resize_mode='Crop and Resize', processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', save_detected_map=True), ControlNetUnit(input_mode=<InputMode.SIMPLE: 'simple'>, use_preview_as_input=False, batch_image_dir='', batch_mask_dir='', batch_input_gallery=[], batch_mask_gallery=[], generated_image=None, mask_image=None, hr_option='Both', enabled=False, module='None', model='None', weight=1, image=None, resize_mode='Crop and Resize', processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', save_detected_map=True), ControlNetUnit(input_mode=<InputMode.SIMPLE: 'simple'>, use_preview_as_input=False, batch_image_dir='', batch_mask_dir='', batch_input_gallery=[], batch_mask_gallery=[], generated_image=None, mask_image=None, hr_option='Both', enabled=False, module='None', model='None', weight=1, image=None, resize_mode='Crop and Resize', processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', save_detected_map=True), False, 7, 1, 'Constant', 0, 'Constant', 0, 1, 'enable', 'MEAN', 'AD', 1, False, 1.01, 1.02, 0.99, 0.95, False, 0.5, 2, False, 256, 2, 0, False, False, 3, 2, 0, 0.35, True, 'bicubic', 'bicubic', False, 0, 'anisotropic', 0, 'reinhard', 100, 0, 'subtract', 0, 0, 'gaussian', 'add', 0, 100, 127, 0, 'hard_clamp', 5, 0, 'None', 'None', False, 'MultiDiffusion', 768, 768, 64, 4, False, False, False, False, False, 'positive', 'comma', 0, False, False, 'start', '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, False, False, False, 0, False, [], 30, '', 4, [], 1, '', '', '', '') {}
    Traceback (most recent call last):
      File "/stable-diffusion-webui-forge/modules/call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
    TypeError: 'NoneType' object is not iterable

---

When I deactivate the extension everything works fine.

[Haoming02]

Could you give these a try first?

https://github.com/Haoming02/sd-forge-couple#typeerror-nonetype

[ostap667inbox]

There is no 'Discussions' tab in this repository so I'll ask here. Are there any recommendations on how to use SD Forge Couple and ControlNet together? I'm particularly interested in how individual objects can be adjusted in ControlNet.

For example, I've made a zone markup of a coastal landscape where I want to represent a spaceship in the sky. When I use only SD Forge Couple everything is fine:

SD Forge Couple mapping:

Prompt:

old intricate curvy tree, 
beautiful sea, water, waves, 
space ship flying in the sky, 
blue sky with beautiful fluffy white clouds, 
rock, a picturesque cliff overhanging the sea, 

So I needed to represent a specific type of spaceship. I took his image placed on a blank image in the desired area and I want to use ControlNet to influence the generation, so that the ship turned out to be the kind I need:

However, no matter which ControlNet model I choose, the result is very bad. SD Forge Couple and ControlNet interfere with each other. I get a deformed or distorted, or very pale and noisy result if I reduce the Control Weight in ControlNet. But if I set Control Weight equal to 1 I get only images of the ship, and also severely damaged because SD Forge Couple interferes with ControlNet. Something like this or this:

[ostap667inbox]

...I only have one way that works. Take a pre-generated image, place a spaceship on it and use ControlNet:

The result is already better (ControlNet Canny used in this case):

However, I would like to get many different images to work with, so that ControlNet only affects the spaceship image and the other objects are different. This method does not allow this. Any ideas?

[ostap667inbox]

There's a great saying, "describe a question in detail and you'll realize it already contains the answer."

I had completely forgotten that ControlNet has masks. You can use an image of a spaceship and prepare a mask of the same shape for it. And the problem is solved: ControlNet will not affect other parts of the image.

There are still some problems from the fact that outside of the mask some detail can be added to the spaceship because SD Forge Couple describes a bigger rectangular area around it. But still, it's already possible to work with it.

[ramyma]

@ostap667inbox Did you try it with the sdxl IP-Adapter plus CN model with a 0.4~0.6 weight and a mask?

[ostap667inbox]

@ostap667inbox Did you try it with the sdxl IP-Adapter plus CN model with a 0.4~0.6 weight and a mask?

With iP-Adapter the results are much less predictable. There may be nothing in the mask area, or there may be an oddly shaped object, or the mask may not be applied at all.

Therefore, it is better to use those ControlNet model types that more accurately affect detail and shape: Canny, Lineart, Depth, Scribble, SoftEdge:

[ramyma]

You can use canny with ip adapter with masks to have better control.

[ramyma]

Also make sure that forge couple is not throwing any warnings in the console because the extension would fail silently with CN sometimes; that's why I opened this thread to start with.

[Haoming02]

~~So... does it actually work or not?~~ (since I don't use IP-Adapter)

[ramyma]

I want to retry it, it seems like it wasn't working initially, but I remember being able to use it after. I'll confirm and close the issue if that's the case.

[sivaranjansahu]

Can confirm this is not working with IP-Adapter CN. Getting the same error as the OP's, and I use SD1.5. I have tried this fix, with no avail. https://github.com/Haoming02/sd-forge-couple#typeerror-nonetype

[Haoming02]

~~four months later...~~

Hello, please give the latest commit a try.

I was finally testing with ip-adapter-faceid-portrait_sd15 today, and I was able to trigger some errors when Forge Couple is also active.

Pushed a commit that stopped that specific error from happening. Please try if other IP-Adapters are also working now~