ControlNet forward wrapper

[continue-revolution]

Description

This is mainly for processing a large batch of frames through ControlNet, ControlLoRA and T2IAdapter. Since I don't know the best way to do this automatically via some offload method, I may do this on my own via sliding window.

[lllyasviel]

hi i am not sure if they are necessary

can you explain a bit why current unet.add_controlnet_conditioning_modifier is not enough for this?

from my read it seems that controlnet_conditioning_modifier can do exactly same thing without changing the kernel?

[continue-revolution]

def mm_cn_forward(forward_fn, cn_type, *args):
    controls = []
    for i in range(0, args[0].shape[0], 2 * self.ad_params.batch_size):
        current_args = (arg[i:i + 2 * self.ad_params.batch_size].to(get_torch_device())
                        if type(arg) == torch.Tensor else arg for arg in args)
        current_ctrl = forward_fn(*current_args)
        if len(controls) == 0:
            controls = [[c.cpu() if type(c) == torch.Tensor else c] for c in current_ctrl]
        else:
            for i, c in enumerate(current_ctrl):
                controls[i].append(c.cpu() if type(c) == torch.Tensor else c)
    for i in range(len(controls)):
        if type(controls[i][0]) == torch.Tensor:
            controls[i] = torch.cat(controls[i], dim=0)
        else:
            controls[i] = controls[i][0] # should be None for T2I Adapters
    return controls

can you do this with the conditioning modifier? If yes, please instruct me how to do it. Otherwise, please just add it.

when sliding window happens in unet forward wrapper, controlnet forward has been finished. this is actually more ideal than sd-webui-controlnet because we don't have to do overlap for cn which is not necessary. but if we don't split forward then it is very easy to OOM for people who want hundreds of frames

[lllyasviel]

thanks let me look at it

[continue-revolution]

also the cond hint itself can sometimes oom for some people and we can't load them all to gpu sometime/

if u have a better way to handle it then that's awesome, otherwise this is at lease use-able.

[continue-revolution]

i also need to modify https://github.com/lllyasviel/stable-diffusion-webui-forge/blob/main/extensions-builtin/sd_forge_controlnet/scripts/controlnet.py#L317 because I want to cap the index to not exceed the max of images across each batch folder. It is easy for me to add a wrapper I'm already familiar enough with ur code.

after this is code, this pr should be ready.

[lllyasviel]

we added https://github.com/lllyasviel/stable-diffusion-webui-forge/commit/a1670c536d5248f2a03b0dae30bb7b10281cdd2f unet.set_controlnet_model_function_wrapper

Feel free to close this PR and open a new one for more things, or just remove duplicated parts in this PR

[continue-revolution]

thanks. i will modify this pr and ping u once it's done