WIP: Add a Vulkan HW interop

[philipl]

This set of changes adds an HW interop (hwdec in our terminology, but there's no decoding going on here) to allow Vulkan frames produced by ffmpeg filters (and eventually vulkan video decoding) to be consumed directly by vo_gpu and vo_gpu_next without any reuploading or copying etc.

• [x] Pass full ra_ctx to hwdecs - needed to fully initialise the Vulkan hwdevice
• [x] Basic implementation of the HW interop
• [x] Replace use of the libplacebo AVFrame based wrapper with code that directly works with the mp_image
• [x] Fix validation errors due to not syncing layout when returning frames to ffmpeg
• [x] Implement a synchronised wrapper for VkQueue operations as we issue commands from both decode and vo threads
• [x] Provide a way to hwupload existing hw frames (cuda/vaapi) to Vulkan. We aren't smart enough to do this today
• [x] Don't require libplacebo v4.200+.
• [x] Add waf support
• [x] Add the right conditionals so the code builds with older ffmpeg and Vulkan SDK
• [x] Add hw->hw upload support to format filter
• [x] Add hw->hw map support to format filter

Testing so far:

• nvidia works with hwupload to vulkan
• intel works with hwmap to vulkan
• amd works with hwmap to vulkan

[philipl]

Example command lines:

mpv --hwdec=nvdec --vf=lavfi=[hwupload,vflip_vulkan]:hwdec_interop=vulkan

or

mpv --hwdec=vaapi --vf=lavfi=[hwmap,vflip_vulkan]:hwdec_interop=vulkan

will decode in hardware, GPU copy/map to vulkan, do filtering and then directly output the vulkan images.

[haasn]

FFmpeg is in the process of gaining lock_queue/unlock_queue functions which will solve the queue threading issue (*), in addition to lock_frame/unlock_frame (which we will need to put inside map/unmap respectively)

(*) We will need a new libplacebo API though to forward these to the imported vulkan context

[cyanreg]

I know you're in a rush to merge this, and sorry you had to spend so much work, but could you wait a day or two so I can merge what's needed in ffmpeg? That way you wouldn't have to maintain that less than nice code.

[philipl]

Sure, I can wait. I still need to sort out the remaining open items. Thanks for your work on the ffmpeg side!

[Obegg]

This PR close this issue: https://github.com/mpv-player/mpv/issues/9675

[cyanreg]

No, it doesn't.

[Obegg]

No, it doesn't.

The issue is a request for a vulkan hwdec, this is the PR that adds it.

[haasn]

No, it doesn't.

The issue is a request for a vulkan hwdec, this is the PR that adds it.

Don't we still need extra code to hook up the --hwdec option?

[philipl]

There are at least four things we need for working vulkan video decode acceeleration

• Drivers that actually implement it. nvidia hasn't included it in any non-developer driver release and the mesa driver work has only just begun
• ffmpeg hwaccels. @cyanreg has been working on this and it's close to done
• The hw interop that can pass VkImages to the vo without conversions. That's this change
• libplacebo changes to ensure we request whatever the necessary video extensions are and set up the video queues

[philipl]

I've done a bunch of refinement to the existing patch set and I'm pretty satisfied with it. The main thing right now is ideally I wouldn't push the ugly wrapper code for locking VkQueue access, and wait for the lock/unlock helpers to land in ffmpeg and libplacebo.

The ra and filter changes can land before the actual vulkan interop, and maybe that's the best way to move things forward.

[philipl]

Added HW->HW upload support to the format filter so now you can do:

mpv --hwdec=nvdec --vf=format=fmt=vulkan,vflip_vulkan

[philipl]

Ok, hwmap support in vf_format is now present.

So you can do

mpv --hwdec=vaapi --vf=format=fmt=vulkan,vflip_vulkan

on Intel, and presumably AMD.

[laichiaheng]

Hasn't it been merged to master branch yet?

[GreyXor]

thanks, Is there any need to test things here?

[philipl]

You can try it out if you want. I don't have AMD hardware so testing there is always useful.

But right now we're blocked on getting locking primitives into ffmpeg. @cyanreg has been working on that.

[GreyXor]

Yep i have AMD iGPU vega 8, i'll test that.

[boxerab]

Nice work, hopefully we can get this merged soon.

One issue I found : I ran the branch with the following pipeline: --hwdec=vaapi --vo=vaapi-wayland, which shouldn't require conversion, and yet build_image_converter does create an auto converter in this case.

[cyanreg]

I know, I'm working on https://github.com/cyanreg/FFmpeg/tree/vulkan_decode It has the necessary locking callbacks and supports decoding. libplacebo and mpv still have to be updated to use multiplane images.

[philipl]

Updated to use the new locking primitives. These have been upstream in libplacebo for a while but the ffmpeg work is still not merged, so I cannot do a correct libavutil version check yet.

Note that the ffmpeg code that is going to be merged uses multi-plane images which we cannot currently handle. That requires both WIP changes in libplacebo and a bunch of work in mpv that I've only scratched the surface of (and is not in this series).

I did all my testing with single plane formats (eg: rgba gifs and 12bit 444 content decoded by Intel vaapi to xv36)

This also means cuda vulkan interop doesn't currently work as cuda cannot handle multi-plane images and the new hwcontext_vulkan code doesn't (has lost the ability to?) work with multi-image based frames.

@cyanreg @haasn

[philipl]

I have updated the patch set to disable multiple plane for now. Until we support rendering multiplane images, we can't actually support the vulkan decoder, but we can support uploaded images as in my original patchset, so we disable multiplane for that purpose.

Supporting filters requires a very new vulkan extension, which only radv currently supports.

[philipl]

Indeed. I've tested it and it works.

[haasn]

Libplacebo master supports multiplane images, so what's blocking them?

[philipl]

General status update:

• Multiplane is working great
• Main remaining blockers are:
  • Testing
  • Merging video queue configuration in libplacebo
  • Coming up with a way to test for vulkan features to fail gracefully

[Obegg]

ffmpeg filters (and eventually vulkan video decoding) to be consumed directly by vo_gpu and vo_gpu_next without any reuploading or copying etc.

If I understand correctly - because there is no reuploading or copying etc it's better than any other interop, So should it be the default interop? (I guess not initially in fear of unexpected bugs and other issues that may rise such as users without the correct driver and etc, but surely this would be the default interop at some point?)

[philipl]

If I understand correctly - because there is no reuploading or copying etc it's better than any other interop, So should it be the default interop? (I guess not initially in fear of unexpected bugs and other issues that may rise such as users without the correct driver and etc, but surely this would be the default interop at some point?)

I'm sure there's a point in the future where you'd prefer vulkan when doing hwdec=auto but that's a good way down the line. It's unclear how well this will all work on Windows vs d3d11va or a hypothetical d3d12va, and given that we're unlikely to see support for older codecs, we'd probably need a multi-hwdec fallback mechanism, which doesn't exist today. The priority right now is just getting the code finalised and merged.

[DeadSix27]

Did Lynne post the patch for ffmpeg public ?

[DeadSix27]

Did Lynne post the patch for ffmpeg public ?

https://github.com/cyanreg/FFmpeg/tree/vulkan

Of course I checked every site but his branches, last issue I encountered is:

./libavutil/vulkan.c:338:27: error: 'VK_QUERY_TYPE_VIDEO_ENCODE_BITSTREAM_BUFFER_RANGE_KHR' undeclared (first use in this function)
  338 |         if (query_type == VK_QUERY_TYPE_VIDEO_ENCODE_BITSTREAM_BUFFER_RANGE_KHR) {

any clue on that? I looked through the web to see where it's defined but couldn't really find a proper source for it other than some mentions in the vulkan docs.

[DeadSix27]

I'm not vulkan expert either, so I removed this IF statement and let it pass the compiler checks, which shouldn't cause a problem since vulkan-encode is disabled by default.

Ran into more issues down the line

../video/out/vulkan/context.c:350:10: error: 'struct pl_vulkan_params' has no member named 'enable_video_decode_queue'
  350 |         .enable_video_decode_queue = true,

../video/out/hwdec/hwdec_vulkan.c: In function 'vulkan_init':
../video/out/hwdec/hwdec_vulkan.c:96:57: error: 'const struct pl_vulkan_t' has no member named 'queue_video_decode'
   96 |     device_hwctx->queue_family_decode_index = vk->vulkan->queue_video_decode.index;
      |                                                         ^~
../video/out/hwdec/hwdec_vulkan.c:97:48: error: 'const struct pl_vulkan_t' has no member named 'queue_video_decode'
   97 |     device_hwctx->nb_decode_queues = vk->vulkan->queue_video_decode.count;

EDIT: Oh he has a libplacebo edit too.. only checked his github, hence I missed his fork on gitlab

[philipl]

VK_QUERY_TYPE_VIDEO_ENCODE_BITSTREAM_BUFFER_RANGE_KHR

You need new enough Vulkan headers. The VK_KHR_video_encode_queue extension was added in 1.2.175, although it's unclear whether all packages of the headers include provisional extensions. In practice you want very recent (newer than Dec 2022) headers to cover all the extensions that the ffmpeg code wants.

[DeadSix27]

You need new enough Vulkan headers.

I'm using git master on all of vulkan though thats the thing.. 1.3.245 to be precise.