[Bug]: only half of the results use ControlNet when using even number in 'Batch size' option in txt2img and img2img pipeline when using Guess mode

[C00reNUT]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

When using the ControlNet in img2img pipeline only half of the results use ControlNet.

Example above uses 'Batch size'=4. I do not mean Batch tab, but 'Batch size' option, please see the example below.

Steps to reproduce the problem

1. Go to img2image pipeline with arbitrary image and prompt
2. Use ControlNet with Guess mode and use any even number 'Batch size', set ControlNet weight to high enough number to see the difference, for example 1.5
3. Half of the results won't use ControlNet as reference

What should have happened?

4 controlnet images

Commit where the problem happens

webui: 22bcc7be428c94e9408f589966c2040187245d81 controlnet: 241c05f8c9d3c5abe637187e3c4bb46f17447029

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--xformers

Console logs

no errors, or non-ordinary console logs

Additional information

No response

[Kanareika]

Your total tokens number in positive and negative prompt must be less than 75, or else you get this.

[C00reNUT]

Your total tokens number in positive and negative prompt must be less than 75, or else you get this.

I'm not sure that's the problem, I have done some more testing and this issue happens only when using even numbers, when I'm using 'Batch size'=5 everything works fine, but when I use 'Batch size'=4 or other even number half of the results don't use ControlNet guidance.

Edit: After even more testing I have found that this bug only occurs when using Guess mode and even numbers in 'Batch size'

[barrkel]

I see the same thing when calling via the API. With an even batch size, half the results don't use the control net.

Additionally, the control net has a substantially weaker effect. The controlled images (i.e. half of the batch) obtained by the API can't be obtained using the UI using the same parameters, unless the values are turned right down - e.g. guidance_end of 0.1 or a weight of 0.25 or so.

I spent a few hours today debugging this issue. An observation: during each sample step, the forward wrapper in UnetHook.hook is invoked, and inside it the value for context.shape[0] is equal to the batch_size argument in the API, but when run via the UI, context.shape[0] is 2 * batch_size. It may or may not be related.

[ljleb]

While working on the batch PR I believe I stumbled on this. One thing that seems to trigger it right now is swapping controlnet models, i.e. replace model 0 settings with model 1's and the other way around too. It only happens with batch mode active in my case, so I'm a bit puzzled.

Although I am not sure this is super helpful as the code in the PR changes rapidly, I thought I would leave this here just in case it could be of any use to narrow it down.

[spaciousmind]

weird, I have this same bug crop up when using controlnet. It happens in txt2img for me too.

it seems to be only even numbers effecting it, can confirm happens with batch sizes of 2,4 and 10 . it doesn't happen with batch sizes of 3 or 5.

here's my current extension versions incase it's of use.