sd-webui-deforum

sd-webui-deforum copied to clipboard

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Color coherence is a huge issue in vid2vid mode of Deforum. If you switch it to None, the colors will basically flicker each frame as you see here https://youtu.be/U9lRAyHwBiQ and color coherence on kills the variety of the palette. Color coherence each N steps don't help much it will just delay the flicker.

What I suggest: render a sequence, then grab the histograms of the finished frame and of the previous one at the selected distance. Either interpolate the histograms somehow, or render an interpolation of the start and the end frames with film or rife and then transfer the histograms using cv match_histograms

Proposed workflow

1. Go to coherence section
2. Press 'backpass', enter the reasonable window size
3. See the animation with no color flicker and more creative colors responding to the prompts

Additional information

No response

Are you going to help adding it? If not, tell why.

Sure

possible other solution: work on using every scheduled N frames to pull a new color pallet to allow for the coherence to be done dynamically.

This is somewhat how the image guidance feature works, however it does it for every frame.

Well, it would be better for an animation to base its colors only on prompts, to minimize human input and to make the results more diverse. It does that if the coherence is None, the problem is that the palette shifts too adruptly

I just did a commit that resets color match sample if strength 0. I plan to add a scene change schedule which will have different options, including a strength threshold for scene change. So, you could set it to like 0.3 for color match if you wanted.

Also, there is now color_coherence = Video Input, which allows you to do a color match against each video frame. There is also an option to only grab a video color match every N frames.

I could also add an option the notebook has, which is to simply provide a color match image.

The suggestion to average stuff might be cool on a short video. However, it wouldn't address the issue of colors drifting over time, which is what color coherence is for in the first place. Basically, unless you keep it somewhere, it drifts.

We just need a better engine for color coherence.