Feature Request: Fix frame sync issues

[cdgriffith]

I just tried out this really handy program for the first time, and it is super easy to use and very nicely presented, thank you!

The only issue I am running into is that the results seem way off. The previous tool I was using https://github.com/gdavila/easyVmaf takes care of frame sync issues and would be nice to incorporate such a thing with this tool.

For example, here is the results from the same reference and test file from FFMetrics:

Vs results from easyVAMF (quickly thrown into a matplotlib plot for compare)

It was a direct encode from the source with no framerate changes, so I am not sure what goes wrong with the comparison, but it seems to be a somewhat common issue.

[fifonik]

Thanks.

If I understand correctly, frames sync could be helpful when you are analyzing video created with "faulty" encoder (encoder that do frames skips/duplicates).

I believe, syncing video streams silently is not a good idea as in fact the is an issue in encoded video (frames are not aligned!) However, as an option for quick analysis it makes sense so I might try to implement it in future.

It would be nice if you supply short files (ref & distorted) that will demonstrate the issue (PM with dropbox link). May it is just an stream offset issue that I'm also trying to deal with.

[cdgriffith]

Example files https://drive.google.com/drive/folders/1Fqpt6BnZE8IxD0QNpHK5QDAWAwE6gRrf?usp=sharing (Video is royalty free from https://pixabay.com/)

Encoded with command: ffmpeg-y -i "paint_reference.mp4" -map 0:0 -c:v libx265 -x265-params "bframes=4:b-adapt=2:frame-threads=0:strong-intra-smoothing=0:hdr10_opt=0:aq-mode=0" -crf 26 -preset slow "paint_slow_26_strong-intra-smoothing=0_hdr10_opt=0_aq-mode=0.mkv"

Without forcing fps speed gets same result as FFMetrics: ffmpeg.exe -y -i "paint_slow_26_strong-intra-smoothing=0_hdr10_opt=0_aq-mode=0.mkv" -i "paint_reference.mp4" -lavfi "[0:v][1:v]libvmaf=log_fmt=json:model_path=vmaf_float_v0.6.1.pkl:log_path=details_bad.json:ssim=1:psnr=1" -report -f null - VMAF score: 80.540863

Using the fix for forcing FPS to be the same: ffmpeg.exe -y -i "paint_slow_26_strong-intra-smoothing=0_hdr10_opt=0_aq-mode=0.mkv" -i "paint_reference.mp4" -lavfi "[0:v]fps=fps=30.0[input0_0];[1:v]fps=fps=30.0[input1_0];[input0_0][input1_0]libvmaf=log_fmt=json:model_path=vmaf_float_v0.6.1.pkl:log_path=details.json:ssim=1:psnr=1" -report -f null - VMAF score: 96.768625

[caiusseverus]

I often get files which have output like this and the option to have FFMetrics correct it would be extremely useful. The above example works well, but having to dig the data out manually for several files is somewhat tedious.

[flaeri]

I've had the same experience, but I think its mainly due to mixing containers. At least that was the conclusion I came to, and as long as I keep the distorted and testfile the same container (both mkv or both mp4), it seems to be a non issue for me. Just wanted to mention it.

This issue happened to me before I found this software as well, when I was just doing it manually in ffmpeg.

[ls-milkyway]

Example files https://drive.google.com/drive/folders/1Fqpt6BnZE8IxD0QNpHK5QDAWAwE6gRrf?usp=sharing (Video is royalty free from https://pixabay.com/)

Encoded with command: ffmpeg-y -i "paint_reference.mp4" -map 0:0 -c:v libx265 -x265-params "bframes=4:b-adapt=2:frame-threads=0:strong-intra-smoothing=0:hdr10_opt=0:aq-mode=0" -crf 26 -preset slow "paint_slow_26_strong-intra-smoothing=0_hdr10_opt=0_aq-mode=0.mkv"

Without forcing fps speed gets same result as FFMetrics: ffmpeg.exe -y -i "paint_slow_26_strong-intra-smoothing=0_hdr10_opt=0_aq-mode=0.mkv" -i "paint_reference.mp4" -lavfi "[0:v][1:v]libvmaf=log_fmt=json:model_path=vmaf_float_v0.6.1.pkl:log_path=details_bad.json:ssim=1:psnr=1" -report -f null - VMAF score: 80.540863

Using the fix for forcing FPS to be the same: ffmpeg.exe -y -i "paint_slow_26_strong-intra-smoothing=0_hdr10_opt=0_aq-mode=0.mkv" -i "paint_reference.mp4" -lavfi "[0:v]fps=fps=30.0[input0_0];[1:v]fps=fps=30.0[input1_0];[input0_0][input1_0]libvmaf=log_fmt=json:model_path=vmaf_float_v0.6.1.pkl:log_path=details.json:ssim=1:psnr=1" -report -f null - VMAF score: 96.768625

That's normal behavior ...FPS & number of frames are required to be same (for both reference & distorted) in order to get correct VMAF results....other requirement: Distorted needed to be scaled to the reference frame size. Color matrix & other color related info should be same for proper analysis. The metric does frame by frame analysis....

Hence, force the encoder to use the same frame rate as of the source.....this must be done by the encoder ....if vmaf is the choice for the analysis.

[Vaenir]

.... Hence, force the encoder to use the same frame rate as of the source.....this must be done by the encoder ....if vmaf is the choice for the analysis.

I've been trying to sus out this issue for the past couple of hours while encoding with svt-av1 through ffmpeg. Encoding from mp4 to mkv with all the same commands results in extremely spiky vmaf graphs similar to the green one in OP, that directly align with similar spikes and dips in PSNR value.

Mp4 to mp4 results in very little frame to frame vmaf and psnr variance, but mp4 to mkv results in high variance.

So I think I'm with @flaeri on this one, I'm chalking it up to weird stuff going on between containers, as I've tried setting output fps to be the same as input fps without any change in results.

[cdgriffith]

Just a knowledge share, in the ffmpeg docs for SSIM, they show that there are extra steps needed for handling different container types to compare. So there is a standard way to handle that.

https://ffmpeg.org/ffmpeg-all.html#ssim

ffmpeg -i main.mpg -i ref.mkv -lavfi  "[0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]ssim" -f null -

[fifonik]

I'm not really sure what you are suggesting. FFMetrics is already using the settb and setpts parameters when calling ffmpeg. You can tick the 'Write FFMpeg commands to log' option, calculate PSNR/SSIM and check the log.

[cdgriffith]

Hi @fifonik glad to hear that is part of FFmetrics!

I was just updating this thread as the last information we had here was to set fps to fix miss-match between containers (which requires a further ffprobe command) and I came across that other way to handle it so wanted to share.

Thanks again for the great interface, will close this out!