VFI happens before extend

[gitmylo]

Expected Behavior

A video should get generated, then extended, then interpolated

Actual Behavior

When using the video extend feature and frame interpolation combined. The generated workflow applies the frame interpolation to the first generated video, then uses that to extend.

So when generating 80 frames, then extending 80 frames with interpolation set to 2, the first half of the video is at 50% speed, the second half is normal speed (base model framerate, not interpolated)

Steps to Reproduce

1. Load a model for text to video or image to video at for example 80 frames
2. Enable video extend and add extend:80 in the prompt (or any other valid number of frames)
3. Enable frame interpolation for your video (in image to video or text to video section depending on what's used)
4. Generate, the result will have the first section interpolated, and the second section raw, the output uses the framerate of an uninterpolated video

Debug Logs

Other

Currently video frame interpolation is a separate setting in 2 parts of the UI, one for t2v and one for i2v. It might make more sense to have one setting, for example in advanced video (although that section is only visible with display advanced options). With the interpolation applied in the end instead of between video gen and video extend.

[XinacS]

Yes, please. This would be great! :)

[willhsmit]

Looking into this, there's two places where the Interpolation code is called.

In T2I, the code on step 10 checks whether there's an extend block, and only interpolates if there isn't one. In I2I, the code on step 11 unconditionally interpolates the initial gen whether there's an extend block or not. The extend code on step 12 doesn't attempt any interpolation.

So the right fix is probably for the I2I code to change to look like the T2I code - skipping interpolation if there's an extend block - and then for the extend block to attempt interpolation once after the final extend. Will try to PR this.

[mcmonkey4eva]

That's probably a sufficient short term fix.

I feel like the correct strategy might actually be for neither t2v nor i2v to interpolate at all, and instead they just set a flag somewhere that the content is a video, and then unify the interp params to a single set under Advanced Video, and then that gets its own block that runs after all the other video gen steps happen.