[Issue]: Control: With batch counts > 1, conditioning is only applied to the first image

[lbeltrame]

Issue Description

When running a batch where the number of images to do is > 1, the conditioning is only applied to the first image. The others show little to no conditioning at all.

Steps to reproduce

1. Load a depth image (e.g. from #3059)
2. Set parameters for Control and for the image
3. Ensure that Batch count is > 1
4. Set the CN parameters (in my case, strength 0.65, start 0, end 0.35, Midas XL model)
5. Hit generate

Expected results

Conditioning is applied to all images

Actual results

Conditioning is only applied to the first image, the others appear to contain part of the control image somehow (see examples)

Example images

These use a different depth image as the other issue but the result is the same.

1. Correct image (batch image=1)

2. Wrong image (batch image=2)

Version Platform Description

SD.Next from devl https://github.com/vladmandic/automatic/commit/13a9e0a333cb1863fd5084399fb9bd6b0a79e174 Python 3.10 NVIDIA A4000 (16G) Cloud platform (Paperspace)

Relevant log output

No response

Backend

Diffusers

Branch

Dev

Model

SD-XL

Acknowledgements

• [X] I have read the above and searched for existing issues
• [X] I confirm that this is classified correctly and its not an extension issue

[lbeltrame]

When doing as simpler prompt with batch = 2, ~~I can't see the issue at all with a different prompt...~~ I can see it when setting batch size > 2. Once processing arrives at the third image, some conditioning has been lost. (From the depth image, both hands should be on the hilt, like it happens for the first two images.

As far as I understand, however, the correct pipeline runs here.

[vladmandic]

can you set env variable SD_CONTROL_DEBUG=true and SD_PROCESS_DEBUG=true and upload the log?

[lbeltrame]

Attached (zipped). There are a couple false starts (wrong batch size, forgetting to set the models...) so only the last 5 generations are the actual test. sdnext.log.zip

[lbeltrame]

Can't reproduce for now (and Vlad can't either), closing until I find a workable minimal working example.

[vladmandic]

i'm not sure if there is a problem. take a look at this example with batch=4:

3rd image clearly does not follow same template as the other ones, but from what i can tell, that is just a factor that controlnet ends early - at only 35% so there is a lot of room for randomness left. as soon as i move controlnet end to a higher value, i get full consistency for all images in batch.

[lbeltrame]

Found a minimal reproducer. Using 7th AnimeXL A, with these prompts (Sampler: DPM++ 2M with non-SDE solver, karras scheduler, 24 steps + hires at 0.45, 15 steps).

1girl, original, (solo:1.1), standing ,pointing, hand on hip,

and (worst quality:1.6),(low quality:1.4),(normal quality:1.2),lowres

With this openpose image

And using this model (conditions better than the stock openpose model)

https://huggingface.co/xinsir/controlnet-openpose-sdxl-1.0/tree/main

This is what you get (grid for batches). Notice how the conditioning is slowly lost over images:

Model is 7th Anime Layer A XL (https://huggingface.co/syaimu/7th_Layer_XL/resolve/main/7th_anime_XL-A-fp16.safetensors?download=true).

[lbeltrame]

I forgot to mention, here conditioning is at 100% (1.00), to avoid issues of introducing variability.

[vladmandic]

can you check with latest dev? things have changed recently how batch processing is done when using controlnet.

[vladmandic]

closing due to no updates - if updates are provided, i'll reopen the issue.