Outputted videos are "very slightly" shorter than input videos

[sloganking]

This issue is similar to #23 but I believe a different bug is causing this.

Outputted videos are shorter by usually only a few seconds. If I rip the audio from the source video and give it to the new video, the beginning of the video is synced up with the audio but the video slowly gets more de-synced with the audio until it's end. This issue is more apparent with longer videos. This is due to frames being dropped very rarely.

[sloganking]

This is purely speculation but I wonder how RIFE is handling scene changes. If RIFE were to not generate an interpolated frame when scenes change, but also not add a frame where that interpolated frame should have been, then RIFE would effectively drop one frame per scene change.

[hzwer]

I guess this is related to the previous memory leak, scene changes should be handled well.

[sloganking]

This issue has existed since before the #26 memory leak was introduced. I will update and test again soon.

Can you think of a way to identify and view the frames which were dropped? That might help us diagnose what is causing this issue.

[hzwer]

I think I fix it. The frame rate should be a float number. It shouldn't be rounded. If this problem persists, I hope you can upload a video to help me reproduce it

[sloganking]

My latest run using an input video with 23.976 FPS shows significant improvement. I don't notice the audio getting more de-synced over time. However the video does seem to be offset by a small amount of time (less than a 10th of a second) with the audio. That offset, while initially difficult to perceive, does make things feel a little weird when watching people speak.

Also, assuming that our calculation for how many frames should be in the output video is ( (startingFrames * frameMultiplier) - 1), my output video is missing 2 (or 0.000093%) of frames.

frameMultiplier being 2 to the power of --exp's value.

I will do more testing soon, and I will share a test video if I am able.

[hzwer]

Which code did you run? inference_video.py?

[sloganking]

Yes python3 inference_video.py --exp=2 --video=video.mp4

edit:

and I am checking the frame count of videos using ffmpeg -i video.mp4 -map 0:v:0 -c copy -f null - and looking at the frame= property at the bottom of what is logged.

[hzwer]

Now the input and output are in separate threads, and both support the format of PNG folder. Is it possible to reconfirm this issue and investigate the cause?

[sloganking]

@hzwer The issue is still not fixed. You're welcome to test videos yourself by opening two video players, one containing the original and the other the interpolated video. Start them at the same time, and pause and play either one until the audio is in sync. Then see if one of the videos has scene changes happen before the other.

For some of the interpolated videos that don't have sound added, videoplayers will also state that the interpolated videos have one second less of video playtime.

[SuperMario20]

Yes python3 inference_video.py --exp=2 --video=video.mp4

edit:

and I am checking the frame count of videos using ffmpeg -i video.mp4 -map 0:v:0 -c copy -f null - and looking at the frame= property at the bottom of what is logged.

can confirm that I have the similar issue with a video. However, my issue is the opposite of yours. My video has an unrounded fps of 29.97, but RIFE reads it as 29.970028956549715 FPS, causing the output to have EXTRA frames. This offsets the audio and makes the audio out of sync in the opposite way and became quite noticeable after only 20 seconds of video. I do not understand why it seems to read the FPS as more than it actually is, but this causes it to have more frames and the output video is 1 second longer than the initial file. The issue is not fixed.