Wrong order of the inputs supplied to VMAF

[Meru]

So I know you mentioned somewhere that you use reversed order of the input clips on purpose and just now I went on to clarify if this is the case.

Given a starting clip, I encoded it 2 times with good settings (Subject A) and slightly worse settings (Subject B)

Then I converted all 3 of those into raw yuv and passed to the official VMAF

Since this way everything is clear and there is no confusion on the order of clips (you specifically tell VMAF which file is reference, which is distorted) I can get more or less correct VMAF values for those clips.

Now I perform VMAF calculation on SOURCE clips, prior to them being decoded into YUV and use FFMPEG for that

First I use correct order for VMAF filter (which is reference 1st, distorted 2nd). The results are consistent with Vanilla VMAF and raw YUV files

Second I use INCORRECT order for VMAF filter (distorted 1st, reference 2nd). The resulting VMAF values are noticeable LOWER and are consistent with FFMetrics.

Conclusion: Since you mentioned it somewhere and I also confirmed the order of clips in FFMetrics in not correct, I file this report as a BUG. The reason for that is while your order can still be used co judge perceived quality between say 2 different encodes of the same source, this is NOT a correct application of the VMAF metric! The main purpose of VMAF is to judge a quality of a totally different videos encoded in ANY possible way and still get a consistently pleasing result for a viewer by reaching a VMAF value of around 95. So having lower VMAF values, such as FFMetrics produces leads to overtweaking of the encoder and overkill bitrates to reach desired VMAF values.

[fifonik]

There is no correct order and incorrect order. You can call ffmpeg and supply your files in any order you like. However, in the filter you have to correctly identify what file used for what purpose.

For example, two these commands to exactly the same (notice files supplied in different order and [0:v] and [1:v] are in different places):

ffmpeg -i A.mp4 -i B.mp4 -lavfi [0:v]settb=AVTB[A];[1:v]settb=AVTB[B];[A][B]libvmaf=log_fmt=json -f null -

ffmpeg -i B.mp4 -i A.mp4 -lavfi [1:v]settb=AVTB[A];[0:v]settb=AVTB[B];[A][B]libvmaf=log_fmt=json -f null -

Show you command line that you think is correct and show the command line that ffmetrics issues to ffmpeg that you believe is incorrect. I will be able to continue after that. Without command lines it is just guessings.

[Meru]

Depending on how you supply clips to VMAF, distorted-reference or reference-distorted, you get different VMAF values for the result. There is a one and only correct way - the one that matches the output of the original vanilla VMAF. And the order that always produces higher VMAF values is the correct one because it is consistent with vanilla VMAF

I specifically haven't posted my commandline because a single commandline has three places where it can get switched around

I am not sure how to get commandline from FFMetrics since I wasnt able to make it be saved in a log. Therefore I tested it in action. I'll check again though

For now I can only post a commandline that is correct ffmpeg -i encode -i source -filter_complex "[0:v]setpts=PTS-STARTPTS[distorted];[1:v]setpts=PTS-STARTPTS[reference];[reference][distorted]libvmaf" -f null -

[fifonik]

https://ffmpeg.org/ffmpeg-filters.html#libvmaf-1

Calculate the VMAF (Video Multi-Method Assessment Fusion) score for a reference/distorted pair of input videos. The first input is the distorted video, and the second input is the reference video. ... Example with options and different containers: ffmpeg -i distorted.mpg -i reference.mkv -lavfi "[0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]libvmaf=log_fmt=json:log_path=output.json" -f null -

The first file (distorted.mpg) named as main ([0:v]...[main]) The second file (reference.mkv) named as ref ([1:v]...[ref]) And then they supplied to filter in this order: [main][ref]libvmaf

In your above example you doing opposite (wrong). Sorry.

[fifonik]

As you see, FFMetrics supplying arguments to ffmpeg exactly as per ffmpeg manual.

[Meru]

Exactly. The manual is INCORRECT

ffmpeg -i distorted.mpg -i reference.mkv here distorted.mpg is 0:v and reference.mkv is 1:v they add label main to 0:v and ref to 1:v bur then they supply it as [main][ref]libvmaf and not [ref][main]libvmaf

[fifonik]

The manual is INCORRECT

Well, in this case you should contact ffmpeg team. However, I pretty sure they do that correctly. It cannot be a few years nobody reported the issue.

bur then they supply it as [main][ref]libvmaf and not [ref][main]libvmaf

I did not get it. They supplying arguments exactly as in the 2nd sentence (that I put in bold).

[Meru]

Instead of assuming that whatever me, you, the manual or someone is correct or not you can just test it by comparing to the original VMAF somewhere here on github. and it all gets clear simply because it accepts input by specifically pointing to files and order is irrelevant.

[fifonik]

In programming world the main assumption -- manual is correct.

If YOU want ffmetrics supply this in different order -- just adjust its conf file with command lines you believe is correct. However, I think it is something gives incorrect values in your particular case (such as strange colour space or framerate) and this cause the calculations looks better for you for that case. I might be wrong, but i can only guess without real command lines and real files. Sorry.

[fifonik]

I am not sure how to get commandline from FFMetrics since I wasnt able to make it be saved in a log

Tick the option in menu. Log created aside of executable so you cannot put it in "C:\Program Files". Just create a folder FFMetrics, put move program there, tick the menu option and you should be able to see log.

[fifonik]

I will try to investigate when I have time (leaving for a week on vacation tomorrow). It would be nice if someone else will check it out (using prepared sources in the same colour spaces and ffmpeg without ffmetrics involved).

[Meru]

It is not a framerate or colorspace issue. The thing is I can sit here proving all day that whatever I did is right or wrong. And I could have made a mistake too. But I think I explained the way I tested things in a pretty reproducible way. FFMPEG commandline is a mess with having so much places things can go wrong. So it is better for someone else to check it out as well to remove any possible bias.

But didn't you mentioned on stackoverflow somewhere that you aware the commandline is wrong but you did it because it was convenient for one reason or another?

For me there are no results that look better. There are results that are consistent with original VMAF and the ones that aren't. I don't use FFMetrics at all. I am just pointing out an issue for people that do use it. Because the way things are now it can't be used to compare various encodes of various sources of various resolutions framerates colorspaces and whatever else.

[fifonik]

But didn't you mentioned on reddit somewhere

Not me.

[Meru]

Sorry i corrected my previous post. I was meant to say stakoverflow

this is a quote of you

However, I do have opposite order in my FFmpeg GUI (FFMetrics).

But maybe I misunderstood the meaning of this line. Anyhow it doesn't change much and it has to be re-tested.

[Meru]

anyway it can be retested like so:

1. Get one source raw YUV file
2. encode it to whatever format with bad settings low-ish bitrate
3. decode it back to raw YUV and test in vanilla VMAF against the source then encode both RAW YUVs into FFV1 lossless and run them through FFMetrics
4. see which order supplied is better in line with the results in (3)

Even if the results wouldn't be 100% identical due to whatever possible conversions FFMPEG performs internally, these conversions won't give an error big enough to impact the metric change due to different order

[fifonik]

Yes, I replied on StackOvervlow. However, if I remember correctly, I was talking about different thing: I meant that while in ffmpeg's example files supplied in command line in order distorted then reference, when files supplied to ffmetrics (in command line), the first file is reference and then it can be many distorted: FFMetrics.exe [options] ref.mp4 file1.mp4 [file2.mp4] [file3.mp4] [...] I did not mean that ffmetrics is supplying arguments in that order to ffmpeg.

[fifonik]

For reference, the FFMetrics.conf needed to supply files to libvmaf in reverse order will look something like this (this is a copy/paste from FFMetrics.conf.example with [ref] and [main] swapped at the end of the line):

{
	"Metrics": {
		"VMAF": {
			"LAVFI": "[0:v]{{main-trim}},{{main-settb}},{{main-setpts}},{{main-scale}},{{main-setrange}},{{main-format}}[main];[1:v]{{ref-trim}},{{ref-settb}},{{ref-setpts}},{{ref-scale}},{{ref-setrange}},{{ref-format}}[ref];[ref][main]{{filter}}"
		}
	}
}

[fifonik]

For now I cannot confirm that.

I created a set of bat-files and used them on my guinea pig video source with ffmpeg.exe 7.1 full from gyan and vmaf.exe v3.0.0.

vmaf.exe and ffmpeg.exe VMAF scores matched as per ffmpeg manual:

Archive with bat files I used and source video: https://www.dropbox.com/scl/fi/9tbjmkhf4kzhaz6nse9z0/VMAF.zip?rlkey=4bvv14y2viwh0j72q63nej8m3&dl=1 (~45MB) Download, unpack, put vmaf.exe and ffmpeg.exe into the folder and run bat files one by one based on numeric filename prefix.

FFMetrics follows the same path -- when ref/distorted not swapped -- the value is smaller. When swapped -- it is higher:

I've also checked ffmpeg's sources and have not found issues in the part that prepare and pass streams to vmaf filter -- they are as per manual (distorted first). It is still possible that they swapped inside libvmaf (I have not checked that, but chances of that from my opinion are very slim), but I think the chances are slim.

It would be nice if you check my scripts and point on mistakes I've made (if any).

[fifonik]

So if you believe there are some issues with ffmpeg/ffmetrics and you'd like to continue the investigation (you mentioned that you do not use FFMetrics so you may be not interested in spending any time on this at all) -- please provide re-produceable example (similar to what I did). Thanks.

[fifonik]

Given a starting clip, I encoded it 2 times with good settings (Subject A) and slightly worse settings (Subject B) Then I converted all 3 of those into raw yuv and passed to the official VMAF

In my tests I did it differently (and I believe this is the correct way to do that):

1. I encoded starting clip with good settings (produced referecne)
2. I encoded reference clip (not the source clip as you did) with worse setting (produced distorted)

In your case you cannot claim that clip encoded with better settings is reference as it could have completely different artifacts.

[fifonik]

Closing as I was not able to re-produce the issue and no further feedback from user.