Do you want VMAF?

[Queuecumber]

🚀 Feature

VMAF is a widely used video quality metric developed by netflix, I have a working implementation of it that I can contribute if you want it.

Here's their mess of a codebase: https://github.com/Netflix/vmaf

Problems:

1. Currently there are no video metrics in the library so if you want it we'll have to decide where to put it
2. My metric depends on calling an external library because the python support sucks.

Motivation

This would allow people to evaluate video quality, it's a pretty standard thing to report these days.

Pitch

Here's the code I'm using for this metric with some redactions (it won't run in this form but you could full in the blanks if you wanted). It has both a class and function metric.

vmaf_command = "vmaf --reference {reference}.yuv --distorted {distorted}.yuv -w {width} -h {height} -p 420 -b 8 --json -o {output}"


class VideoMultiMethodAssessmentFusion(Metric):
    def __init__(self):
        super().__init__()
        # TODO support other accumulations I guess

        self.add_state("accumulation", default=torch.tensor(0.0), dist_reduce_fx="sum")
        self.add_state("total", default=torch.tensor(0.0), dist_reduce_fx="sum")

        # TODO check if vmaf binary exists, download it if it doesn't

    def update(self, distorted: Tensor, reference: Tensor):
        vmaf_batch = vmaf_metric(distorted, reference)

        self.accumulation += vmaf_batch
        self.total += distorted.shape[0]

    def compute(self):
        return self.accumulation / self.total


def vmaf_metric(distorted: Tensor, reference: Tensor) -> float:
    frame_size = distorted.shape[-2:]

    vmaf = 0
    with TemporaryDirectory() as d:
        d = Path(d)

        for dis, ref in zip(distorted, reference):
            write_video(dis, d / "distorted", codec="reallyraw")
            write_video(ref, d / "reference", codec="reallyraw")

            check_call(
                vmaf_command.format(reference=str(d / "reference"), distorted=str(d / "distorted"), width=frame_size[1], height=frame_size[0], output=str(d / "report")).split(),
                stderr=subprocess.DEVNULL,
                stdout=subprocess.DEVNULL,
            )

            with (d / "report").open() as f:
                report = json.load(f)

            vmaf += report["pooled_metrics"]["vmaf"]["mean"]

    return vmaf

More Info

I'm not totally happy about the external binary, I may bug netflix to make a proper python package

[SkafteNicki]

Hi @Queuecumber, I think it would be a great contribution to have in torchmetrics as it would be our first metric focusing on video evaluation.

I think we should be able to install vmaf as an python library based on the description here: https://github.com/Netflix/vmaf/blob/master/resource/doc/python.md and then we should have access to the installed package as here: https://github.com/Netflix/vmaf/blob/master/python/vmaf/script/run_vmaf.py what do you think?

[Queuecumber]

I think we should be able to install vmaf as an python library based on the description here

Yes we could try that but how do we specify it as a dependency if it isnt on pypi? That's the only real blocker I have with it

[SkafteNicki]

Installing directly from github is possible. Something like this should do it:

pip install "vmaf @ git+https://github.com/Netflix/vmaf@master#subdirectory=python"

and it would be

vmaf @ git+https://github.com/Netflix/vmaf@master#subdirectory=python

in corresponding requirement.txt file.

[Queuecumber]

let me see if that works, and as long as you're ok with having a git dependency

[SkafteNicki]

let me see if that works, and as long as you're ok with having a git dependency

we already have one :) https://github.com/Lightning-AI/metrics/blob/3bf1491141ef30bf89251f11770cd49d500f3571/requirements/audio.txt#L1-L2

[Queuecumber]

Alright then I'll give this a go and get back to you

[Queuecumber]

The good news is that it seems to install with pip, the bad news is that based on my reading of their code it still requires raw videos to be written to disk.

Will continue looking for a workaround for that.

[Queuecumber]

And more bad news, it looks like everything ends up just calling the binary anyway

https://github.com/Netflix/vmaf/blob/master/python/vmaf/init.py#L130

[Borda]

Binaries are quite tricky to maintain... :(

[Queuecumber]

Yeah I agree, what we need is a wrapper library around libvmaf, which is what I've been investigating. I don't think it's particularly difficult to do from what I saw

Ideally this would be separate from torchmetrics

Unfortunately we're getting into territory where Nvidia may have opinions about me donating that much code so it's not something I can just do without checking with them first

[Borda]

@SkafteNicki, any other thoughts? :rabbit: