how to use a video as an input for controlnet

[PBoy20511]

I'm trying to use a video as an input for controlnet. I wrote a python code that converts my video into a png sequence. Then I want to use a loop to put these pictures one by one into controlnet, but I can’t find the .py file that can input pictures into controlnet. Can anyone help me?

[toyxyz]

This extension includes a ControlNet M2M script that allows direct input of mp4 video files. Multi-ControlNet is also supported. Here's a script I've modified if you'd like to use image sequences instead. https://toyxyz.gumroad.com/l/jydvk

[roryoday]

I implemented this on my repository. It's a hacky solution but it works.

I use the controlnet_fromfile script in txt2img

Here is an example:

--controlnet /canny/0000.png,/openpose/orig_0000.png,/hed/0000.png --prompt example prompt --seed 843052413 --steps 25 --cfg_scale 9 --width 512--height 512

--controlnet /canny/0001.png,/openpose/orig_0001.png,/hed/0001.png --prompt example prompt --seed 843052413 --steps 25 --cfg_scale 9 --width 512--height 512

the controlnet parameter should be a list separated by commas that corresponds to the number of controlnet models you have active. It should be a filepath to the image that you want to use for that model.

After that I just use ffmpeg to make a video from the output.

The results flicker much more than toyxyz's img2img approach and it only supports a batch size of 1.

[marcsyp]

This extension includes a ControlNet M2M script that allows direct input of mp4 video files. Multi-ControlNet is also supported. Here's a script I've modified if you'd like to use image sequences instead. https://toyxyz.gumroad.com/l/jydvk

@toyxyz -- do you have any intention of creating a version of your script for txt2img? Txt2 img results are generally better than img2img in many cases, even with the skip processing checkbox enabled in i2i.

Cheers

[ljleb]

@marcsyp do you think we should remove the skip txt2img processing checkbox? I think we don't need it anymore now that multi-cn batch is implemented.

[ljleb]

Possible that some scripts run into issues communicating with mulci-cn txt2img batch as there is no public API for multi-cn batch at the moment.

[marcsyp]

@ljleb

I am usually of the philosophy that you don't remove features unless there is some tangible benefit to doing so and/or it improves the clarity or UX for the average user. In this case, the check box is an optional feature that is really only enabled by users for an intended purpose -- and though the CN batch features certainly replace the need for what most users were using the checkbox for (to get closer to true t2i in i2i), the results are in fact different from the true t2i batch functionality.

There is a chance that some users out there would rely on this option being available to be able to reproduce the exact "inbetween t2i and i2i" effect that it gives, so I would err on the side of caution and just leave it in there. It's not really hurting anybody to have it as an option, afaict.

Thank you btw for your awesome work on CN Batch functionality -- I have been using controlnet-m2m for this purpose but Batch is soooooo much better for workflow (better interrupt, not having to concatenate images every time I want to choose source images, etc). This ecosystem is really awesome thanks to all the amazing contributors here. Kudos!

Cheers, Marc

[ljleb]

Sure, that makes sense. It adds a little bit of complexity to the code, making it harder to test for whether we are in img2img or txt2img. Additionally, IIUC it also complexifies the user interface with alternative ways of doing the same thing.

There is a chance that some users out there would rely on this option being available to be able to reproduce the exact "inbetween t2i and i2i" effect that it gives, so I would err on the side of caution and just leave it in there. It's not really hurting anybody to have it as an option, afaict.

I am not sure what could cause this discrepancy, are you aware of the specifics? As the checkbox swaps the pipeline with txt2img, I would think it should generate very similar images, if not exactly the same images in both cases.

In any case, I am not pushing for this change, I was just seeking the point of view of an interested party on the topic. Thanks for your input.