Controlnet batch support

[rulyone]

Describe what this pull request is trying to achieve.

Enable ControlNet extension batch support for control images.

Additional notes and description of your changes

Currently there's only batch support for inpainting masks. I simply adapted the following inpainting batch support pull request (https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/7295/files), but for ControlNet control images instead.

This enables using 1 control image per 1 img2img input image, which helps to create videos combining different techniques, as can be seen here as a result of this technique with a live background and a live pianist I created using these changes: https://www.youtube.com/watch?v=ciMeSHmq0hg

This change can only be usable after the following Pull Request is approved in AUTOMATIC1111/stable-diffusion-webui repository: https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/8125

Environment this was tested in

• OS: Linux
• Browser: chrome
• Graphics card: NVIDIA RTX 3090 24GB

Screenshots or videos of your changes

The following changes were made in AUTOMATIC1111/stable-diffusion-webui, but I'll add the screenshot here for context too:

[marcsyp]

This is beautiful. A few requests:

• ability to ignore the img2img source image (essentially replicating what you would get out of txt2img)
• different directories for up to 10 controlnets (to support different sources images for different active control nets)

First point is highest priority for me! Related to:

https://github.com/lllyasviel/ControlNet/issues/171

[ljleb]

ability to ignore the img2img source image (essentially replicating what you would get out of txt2img)

For txt2img batch, I believe there is already an option in the settings "Skip img2img processing when using img2img initial image". Does it cover your use-case?

[demirklvc]

is this now possible? read it in the commits but cant find the function in the web ui

[AbyszOne]

is this now possible? read it in the commits but cant find the function in the web ui

Same here.

[fyoudumas]

This change can only be usable after the following Pull Request is approved in AUTOMATIC1111/stable-diffusion-webui repository: AUTOMATIC1111/stable-diffusion-webui#8125

[AbyszOne]

Ops. Patience. By the way, the different batches per layer is a very relevant update. Even that you can choose which uses batch and which remains static. Simple features that would push the extension to the level of advanced animation. Another would be that the batch of nets runs maintaining the same base image in img2img, although that can already be done in a rudimentary way. By the time these features are added, the latest corridor crew video is going to be outdated.

[marcsyp]

ability to ignore the img2img source image (essentially replicating what you would get out of txt2img)

For txt2img batch, I believe there is already an option in the settings "Skip img2img processing when using img2img initial image". Does it cover your use-case?

Yes! Thank you.

[marcsyp]

Ops. Patience. By the way, the different batches per layer is a very relevant update. Even that you can choose which uses batch and which remains static. Simple features that would push the extension to the level of advanced animation. Another would be that the batch of nets runs maintaining the same base image in img2img, although that can already be done in a rudimentary way. By the time these features are added, the latest corridor crew video is going to be outdated.

Yes, I would love to see a "per layer" batch directory, and I hope others do too -- would be awesome, particularly with the new Guidance start-stop functionality, which essentially allows you to sequence controlnets.

[AbyszOne]

Now imagine a box that allows you to program each batch by frames, and another a prompt by layer, and a mask by layer. That's probably what deforum will become, although it's been way behind lately.

[rulyone]

For multiple layers of batch control, if auto1111 PR is approved, we could modify here in the extension to read subfolders with a specific naming eg: layer0, layer1, etc if they're found inside the main folder. That would allow to not add so many folders in the options to the user, and use only 1, but with a predefined structure as mentioned, eg:

AUTO1111 folder config input dir:

/home/diffusing/controlimages

and inside that, we could have /home/diffusing/controlimages/layer00/img001.png /home/diffusing/controlimages/layer00/img002.png /home/diffusing/controlimages/layer00/imgxxx.png /home/diffusing/controlimages/layer01/img001.png /home/diffusing/controlimages/layer01/img002.png /home/diffusing/controlimages/layer01/imgxxx.png /home/diffusing/controlimages/layer02/img001.png /home/diffusing/controlimages/layer02/img002.png /home/diffusing/controlimages/layer02/imgxxx.png

for 3 layers. If no subfolder is found, then use the same folder for all layers.

I could work on this commit as soon as the AUTO1111 PR is approved.

[AbyszOne]

Simple and effective. I like your style. Let's dig a little then. 1- What would happen if some folders have less number of frames? Last loop or disabled? 2- If they have only one? Loop or disable?

It would be a relevant parameter, although salvageable. Now, a much bigger question. Can you create a sub structure of images too? I mean that it respects the parity between names for scene programming. For example: Layer0/F1.jpg Layer0/F2.jpg Layer0/F4.jpg

Layer1/F2.jpg Layer1/F3.jpg Layer1/F4.jpg

And that the batch respects the interaction of each frame with its respective pair, allowing you to activate and deactivate all the frames of all the layers at will without the need for further modification of the gui. (More in new comments).

[ljleb]

Maybe we could let users chose between different strategies for handling different-sized batches for each layer/unit. Here are other suggestions from this comment:

1. handle similarly to zip python function, i.e. "zip" together all batch dirs, and strip the resulting list to the length of the batch dir with the least number of images. generates n images, with n = number of images in the smallest batch dir
2. multiplex batch dirs, i.e. for n given batch dirs, for each image in each batch dir, generate 1 image. generates N1 * N2 * N3 *... * Nx batch images, with x = number of batch dirs

Again, I'm not sure multiplex is a good idea. Maybe it would make sense to be able to select multiplex for some control units while using zip with the rest? In a sense, this allows to batch process multiple times other batches.

The name pairing strategy described by @AbyszOne could be an additional parameter to the multi-controlnet batch handling. You could let the user decide the strategy to determine how each image of multiple batches are paired together. Name pairing is an idea, but also lexicographic order (disregarding whether 2 images in different batches have the same name) could be another for example.

[AbyszOne]

Great. I have updated the parity strategy in That thread.

[marcsyp]

I love the sublayer approach -- a few things to note, for use cases/fallbacks:

• If images exist in the parent directory AND in sublayers, parent director images are ignored.
• if the user has more controlnets activated than subdirectories exist, there could be two options: "cycle" (goes back to the first subdirectory) and "use last" (uses the last existing control directory repeatedly until all controlnets are satisfied). This would be useful for both images that need multiple processes on each image, and for situations where you need one controlnet for one image, and multiple controlnets for another.

[joshdstanton]

ability to ignore the img2img source image (essentially replicating what you would get out of txt2img)

For txt2img batch, I believe there is already an option in the settings "Skip img2img processing when using img2img initial image". Does it cover your use-case?

Yes! Thank you.

Just what I was looking for. Thanks!

[autojadame]

For multiple layers of batch control, if auto1111 PR is approved, we could modify here in the extension to read subfolders with a specific naming eg: layer0, layer1, etc if they're found inside the main folder. That would allow to not add so many folders in the options to the user, and use only 1, but with a predefined structure as mentioned, eg:

AUTO1111 folder config input dir:

/home/diffusing/controlimages

and inside that, we could have /home/diffusing/controlimages/layer00/img001.png /home/diffusing/controlimages/layer00/img002.png /home/diffusing/controlimages/layer00/imgxxx.png /home/diffusing/controlimages/layer01/img001.png /home/diffusing/controlimages/layer01/img002.png /home/diffusing/controlimages/layer01/imgxxx.png /home/diffusing/controlimages/layer02/img001.png /home/diffusing/controlimages/layer02/img002.png /home/diffusing/controlimages/layer02/imgxxx.png

for 3 layers. If no subfolder is found, then use the same folder for all layers.

I could work on this commit as soon as the AUTO1111 PR is approved.

How is it going with this? Can I help with anything?

[rulyone]

For multiple layers of batch control, if auto1111 PR is approved, we could modify here in the extension to read subfolders with a specific naming eg: layer0, layer1, etc if they're found inside the main folder. That would allow to not add so many folders in the options to the user, and use only 1, but with a predefined structure as mentioned, eg:

AUTO1111 folder config input dir:

/home/diffusing/controlimages

and inside that, we could have /home/diffusing/controlimages/layer00/img001.png /home/diffusing/controlimages/layer00/img002.png /home/diffusing/controlimages/layer00/imgxxx.png /home/diffusing/controlimages/layer01/img001.png /home/diffusing/controlimages/layer01/img002.png /home/diffusing/controlimages/layer01/imgxxx.png /home/diffusing/controlimages/layer02/img001.png /home/diffusing/controlimages/layer02/img002.png /home/diffusing/controlimages/layer02/imgxxx.png

for 3 layers. If no subfolder is found, then use the same folder for all layers.

I could work on this commit as soon as the AUTO1111 PR is approved.

How is it going with this? Can I help with anything?

Waiting auto1111 webui approval. He's been MIA the past 3 weeks maybe vacations. If this doesnt get approved on that side, any effort will be dismissed if going on this direction.

[SeanBannister]

According to this it's now possible?

[AbyszOne]

According to this it's now possible?

Not related.

[marcsyp]

Note to others using the solution above, that the "skip img2img" checkbox does help simulate txt2img batching, but is not a replacement for true batching with txt2img. There is still some pollution coming from the img2img process that does not exist with txt2img... my interim solution has been to make an mp4 with the images I want to batch and use m2m script in txt2img while a true batching solution is being developed.