Vulkan Video Decoding: Usage Guide and FAQ

On the 28th of May, we reached the significant milestone of finally merging all the required functionality into ffmpeg, libplacebo, and mpv to do end-to-end Vulkan video decoding and presentation. Although the functionality is now all there, we still have a complex landscape in terms of what is supported on what hardware and with what drivers. This document attempts to lay out all the requirements and limitations, so you have a chance of successfully using the feature.

Why should I even care about Vulkan video decoding?

It's a fair question. Right now, the actual functionality you gain access to is not terribly different from what you would get with existing video decoding APIs that ffmpeg and mpv already support. However, Vulkan video decoding has the potential to be a credible cross-vendor, cross-platform API that is well supported, and can work efficiently with Vulkan based filtering and post-processing. In the short term, there will be rough edges, and only a narrow set of supported codecs (only H.264 and H.265 are standardised today, with AV1 still a work in progress), but in the future, it will hopefully be the most performant and capable way to do hardware decoding and processing.

Software Requirements

We have finally reached the point where all the required components have made official releases. Ensure you are running with the following releases or newer:

ffmpeg: 6.1
libplacebo: v6.292.0
mpv: 0.37.0

These requirements mean you will likely have to compile everything for yourself (especially on Linux), but Shinchiro's windows builds appear to be functional.

Hardware and Driver Requirements

Intel on Linux
- Mesa: mesa-23.2.1
- A GPU supported by the ANV driver
- An MR is open for support in the hasvk driver
AMD on Linux
- Mesa: mesa-23.2.1
- A GPU supported by the RADV driver
- The AMDVLK driver is not known to support video decoding
Nvidia on Linux or Windows
- Drivers >= 535.xx
- Any GPU supported by those drivers
AMD on Windows
- Drivers >= 23.9.3
- RDNA 1/2/3 GPUs
Intel on Windows
- Unclear and untested
- There is evidence that drivers from at least 101.4311 expose the video decoding extensions
- No idea what hardware is supported
MacOS (for the record)
- Although Vulkan on MacOS is supported via MoltenVk, this does not include the decoding extensions, and probably never will. Just keep using videotoolbox.

Usage

The basic command line arguments:

mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan

Depending on your system configuration, you may additionally need to force the right gpu-context:

Windows: --gpu-context=winvk
Linux X11: --gpu-context=x11vk
Linux Wayland: --gpu-context=waylandvk

Capabilities

Global
- The Vulkan Video Decoding specs only cover H.264, H.265, and AV1 as a work-in-progress. Older codecs will require you to use your traditional hardware decoding API (VAAPI, nvdec, dx11va, etc) for now, and possibly forever - depending on whether effort is put into exposing older codecs.
Intel
- You must set ANV_VIDEO_DECODE=1 in your environment to expose video decoding
- Both H.264 and H.265 are supported
- VK_EXT_descriptor_buffer support, required for Vulkan filters, still needs to be merged
AMD
- You must set RADV_PERFTEST=video_decode in your environment to expose video decoding
- Both H.264 and H.265 are supported
- Vulkan filters are supported
Nvidia
- Both H.264 and H.265 are supported
- Vulkan filters are supported

May 29 '23 17:05 philipl

AMD on Windows appears to be 5000/6000 series according to https://www.amd.com/en/support/kb/release-notes/rn-rad-win-22-11-2-vlk-video-code-decode (possibly 7000 works?). It doesn't work on Polaris: 0.339][e][ffmpeg/video] h264: Device does not support the VK_KHR_video_decode_queue extension!

May 29 '23 17:05 Jules-A

After bc28dce303a4abcc8ec2f1fd941bc1bfd02e6f79, you need to patch your Mesa with this PR to get AV1 decoding on radv.

May 29 '23 21:05 llyyr

Reliably crashes driver for me with this sample: https://0x0.st/HbHJ.mkv Other files seem to work for whatever reason (hwdec beeing used confirmed). Latest git-master of mpv, libplacebo, ffmpeg and radv on 6700 xt.

May 30 '23 00:05 aufkrawall

Sounds like you probably need to file a mesa issue for that - and even if it was an mpv issue, please open a separate one. This FAQ isn't the place to report it.

May 30 '23 00:05 philipl

@aufkrawall AMD currently needs https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23227 to avoid crashes, which also fixes quite a lot of bugs

May 30 '23 01:05 cyanreg

 (+) Video --vid=1 (*) (hevc 3828x1592 23.976fps)
 (+) Audio --aid=1 --alang=eng (*) (eac3 6ch 48000Hz)
     Subs  --sid=1 --slang=eng (subrip)
     Subs  --sid=2 --slang=eng 'SDH' (subrip)
     Subs  --sid=3 --slang=fre (subrip)
[vo/gpu-next/libplacebo] vk->CreateDevice(vk->physd, &dinfo, PL_VK_ALLOC, &vk->dev): VK_ERROR_OUT_OF_HOST_MEMORY (../../../../../src_packages/libplacebo/src/vulkan/context.c:1308)
[vo/gpu-next/libplacebo] Failed creating logical device!
[vo/gpu-next/libplacebo] Failed initializing vulkan device
[vo/gpu-next] Failed initializing any suitable GPU context!
Error opening/initializing the selected video_out (--vo) device.
Video: no video

what does this error mean, my device has amd and i used the following config mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=winvk mpv was working fine with gpu-next and gpu-api=vulkan till yesterdays' shinchiro release. todays release gave me this error maybe @shinchiro needs to bump libplacebo and ffmpeg version as per required?

May 30 '23 03:05 alakittty

AMD on Windows Special 22.11.2 drivers with video decoding support Hardware supported by those drivers (documented as 5000/6000 series. Apparently not newer 7000 series)

I don't know if that specific driver is required, they may have been included in 23.2.1 where they added other extensions, the driver notes aren't clear. 7000 series wasn't launched when the 22-11-2 Vulkan video drivers were released so that driver doesn't officially support them. If they did include the video extensions in newer drivers then they may support 7000 series but someone with those GPUs will need to test.

May 30 '23 11:05 Jules-A

Yeah, I've reported to nvidia months ago that their descriptor buffer implementation is broken, but they haven't fixed it yet. If you want filters working, ask nvidia to fix it.

May 30 '23 13:05 cyanreg

I found this video will white screen when using ffmpeg vulkan decoding, SW decoding or nvpro vk_video_decoder have no problem, I don't know if this is also NVIDIA's problem (also already reported to NVIDIA) test.mp4

Github claims you attached an audio-only file...

May 30 '23 16:05 philipl

That file is also broken on anv, it's glitchy and the decode is very slow while utilizing 100% of Render/3D engine.

May 30 '23 17:05 llyyr

That file is also broken on anv, it's glitchy and the decode is very slow while utilizing 100% of Render/3D engine.

It's an HEVC file. My experience on ANV is that any HEVC playback pegs the GPU at 100%

May 30 '23 17:05 philipl

Also:

[ffmpeg/video] hevc: First slice in a frame missing.

This is a highly questionable sample.

May 30 '23 17:05 philipl

Will this work with hasvk in the future? I tried using it through wine (I dont want to have to compile everything) and got this error [ffmpeg/video] h264: Device does not support the VK_KHR_video_decode_queue extension! even though the output of vulkaninfo is:

vulkaninfo | grep video                                                                     ✔  10s  
	VK_KHR_video_decode_h264                    : extension revision 8
	VK_KHR_video_decode_queue                   : extension revision 7
	VK_KHR_video_queue                          : extension revision 8
			videoCodecOperations:
			videoCodecOperations: count = 1

May 31 '23 21:05 lavilao

NVIDIA 1650 Super, Tested on windows with: mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=winvk 5 videos played OK.

This video has problem: https://4kmedia.org/sony-whale-in-tonga-hdr-uhd-4k-demo/ Switched back to hwdec=nvdec, the problem is gone.

May 31 '23 22:05 CharlesMengCA

for fedora, you need to edit mesa.spec and rebuild with codecs enabled -D video-codecs=h264dec,h264enc,h265dec,h265enc,vc1dec, codecs is disabled by default.

https://github.com/rpmfusion-infra/fedy/issues/110

https://themaister.net/blog/2023/01/05/vulkan-video-shenanigans-ffmpeg-radv-integration-experiments/

https://forum.endeavouros.com/t/compiling-mesa-22-2-with-codec-support-the-easy-way/30390

Jun 01 '23 15:06 24fpsDaVinci

works well with intel 11th gen igpu

for those @lavilao with [ffmpeg/video] h264: Device does not support the VK_KHR_video_decode_queue extension! make sure to enable mesa video codecs as stated here: https://themaister.net/blog/2023/01/05/vulkan-video-shenanigans-ffmpeg-radv-integration-experiments/ https://forum.endeavouros.com/t/compiling-mesa-22-2-with-codec-support-the-easy-way/30390

(small bug: pressing Ctrl+H will toggle vaapi instead of vulkan)

I enabled them. Check the vulkaninfo log, it shows the extension but mpv does not recognizes it. This is what I used to compile it

meson .. --reconfigure -D b_lto=true -D b_pgo=off -D platforms=x11,wayland -D buildtype=release -D prefix="$HOME/anv-master-video" --libdir="$HOME/anv-master-video/lib" -D b_ndebug=true -D gallium-drivers= -D vulkan-drivers=intel_hasvk -D gles1=disabled -D gles2=disabled -D opengl=false -D video-codecs=h264dec,h264enc,h265dec -D vulkan-beta=true

Jun 01 '23 21:06 lavilao

@lavilao i assume you're using Shinchiro's windows build? I haven't tried using mpv through wine, you may need to recompile.

Jun 01 '23 22:06 24fpsDaVinci

(+) Video --vid=1 (*) (hevc 3840x1608 24.000fps)
(+) Audio --aid=1 --alang=chi (*) (eac3 6ch 48000Hz)
    Audio --aid=2 --alang=chi (*) (aac 2ch 44100Hz)
[vo/gpu-next/libplacebo] Missing device feature: dynamicRendering
[vo/gpu-next/libplacebo] Vulkan device does not support all required features!
[vo/gpu-next/libplacebo] Failed creating logical device!
[vo/gpu-next/libplacebo] Failed initializing vulkan device
[vo/gpu-next] Failed initializing any suitable GPU context!
Error opening/initializing the selected video_out (--vo) device.
Video: no video

what does this error mean, my device has intel gpu, and i used the following config mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=winvk using the latest shinchiro release.

Jun 02 '23 13:06 zhengqwe

Intel on Windows does not support Vulkan hwdec, read the OP

Jun 02 '23 13:06 llyyr

Might be mistaken, but this appears to have allowed zero copy hwacceleration on my polaris on radv. seems like preformance is a bit better too

Jun 02 '23 18:06 Quackdoc

@lavilao i assume you're using Shinchiro's windows build? I haven't tried using mpv through wine, you may need to recompile.

I recompiled on a arch distrobox and the good news is it detects it. The bad news are that the logs show a driver bug:

 (+) Video --vid=1 (*) (h264 1278x720 30.000fps)
 (+) Audio --aid=1 --alang=eng (*) (opus 2ch 48000Hz)
[vo/gpu-next/wayland] GNOME's wayland compositor lacks support for the idle inhibit protocol. This means the screen can blank during playback.
MESA-INTEL: warning: ../src/intel/vulkan_hasvk/anv_formats.c:784: FINISHME: support more multi-planar formats with DRM modifiers
Using hardware decoding (vulkan).
AO: [pulse] 48000Hz stereo 2ch float
VO: [gpu-next] 1278x720 vulkan[nv12]
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:358)
[vo/gpu-next] Failed presenting frame!
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:358)
[vo/gpu-next/libplacebo] Retrieving query pool results: VK_ERROR_DEVICE_LOST (../src/vulkan/gpu.c:103)
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:358)
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:358)
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:358)
[vo/gpu-next/libplacebo] Retrieving query pool results: VK_ERROR_DEVICE_LOST (../src/vulkan/gpu.c:103)
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:358)
[vo/gpu-next/libplacebo] Failed holding swapchain image for presentation
[vo/gpu-next] Failed presenting frame!
AV: 00:00:00 / 00:08:13 (0%) A-V:  0.000
[ffmpeg] av_log callback called with bad parameters (NULL AVClass).
[ffmpeg] This is a bug in one of Libav/FFmpeg libraries used.
[ffmpeg] Unable to submit command buffer: VK_ERROR_DEVICE_LOST
[ffmpeg/video] h264: get_buffer() failed
[ffmpeg/video] h264: decode_slice_header error
[ffmpeg/video] h264: no frame!
Error while decoding frame (hardware decoding)!
[ffmpeg] av_log callback called with bad parameters (NULL AVClass).
[ffmpeg] This is a bug in one of Libav/FFmpeg libraries used.
[ffmpeg] Unable to submit command buffer: VK_ERROR_DEVICE_LOST
[ffmpeg/video] h264: get_buffer() failed
[ffmpeg/video] h264: decode_slice_header error
[ffmpeg/video] h264: no frame!
Error while decoding frame (hardware decoding)!
[ffmpeg] av_log callback called with bad parameters (NULL AVClass).
[ffmpeg] This is a bug in one of Libav/FFmpeg libraries used.
[ffmpeg] Unable to submit command buffer: VK_ERROR_DEVICE_LOST
[ffmpeg/video] h264: get_buffer() failed
[ffmpeg/video] h264: decode_slice_header error
[ffmpeg/video] h264: no frame!
Error while decoding frame (hardware decoding)!
[ffmpeg] av_log callback called with bad parameters (NULL AVClass).
[ffmpeg] This is a bug in one of Libav/FFmpeg libraries used.
[ffmpeg] Unable to submit command buffer: VK_ERROR_DEVICE_LOST
Falling back to software decoding.
[ffmpeg/video] h264: co located POCs unavailable
[ffmpeg/video] h264: co located POCs unavailable
AV: 00:00:00 / 00:08:13 (0%) A-V:  0.000
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:358)
[vo/gpu-next/libplacebo] Retrieving query pool results: VK_ERROR_DEVICE_LOST (../src/vulkan/gpu.c:103)
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:358)
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:358)
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:358)
[vo/gpu-next/libplacebo] Failed holding swapchain image for presentation
[vo/gpu-next] Failed presenting frame!
AV: 00:00:00 / 00:08:13 (0%) A-V:  0.000

Exiting... (Quit)

Jun 02 '23 20:06 lavilao

Ah, hasvk (old Intel GPUs) has a PR to implement video decoding that hasn't been merged yet: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21183

Jun 02 '23 21:06 cyanreg

@cyanreg thats the version I am using

Jun 02 '23 23:06 lavilao

(+) Video --vid=1 (*) (hevc 3840x1608 24.000fps)
(+) Audio --aid=1 --alang=chi (*) (eac3 6ch 48000Hz)
    Audio --aid=2 --alang=chi (*) (aac 2ch 44100Hz)
[vo/gpu-next/libplacebo] Missing device feature: dynamicRendering
[vo/gpu-next/libplacebo] Vulkan device does not support all required features!
[vo/gpu-next/libplacebo] Failed creating logical device!
[vo/gpu-next/libplacebo] Failed initializing vulkan device
[vo/gpu-next] Failed initializing any suitable GPU context!
Error opening/initializing the selected video_out (--vo) device.
Video: no video
what does this error mean, my device has intel gpu, and i used the following config mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=winvk using the latest shinchiro release.
Make sure your GPU is Gen12 or higher (TigerLake+), and update the driver to 4311 or higher

i use i3-10110u's gpu. so I still use d3d11va now. it works well.

Jun 03 '23 11:06 zhengqwe

I usually use vo=dmabuf-wayland but noticed that it's causing a segfault when vulkan interop is compiled in (even when it's not being used). Seems like when using dmabuf-wayland it attempts to run through loading every hwdec option even if one is explicitly specified.

Happy to use gpu-next but thought I should note this bug. But also curious if this will work with dmabuf-wayland in the future?

Jun 04 '23 00:06 kkost

Someone make this into a Wiki entry so I can stop get notifications. Clearly not shithub issue material.

Jun 04 '23 01:06 garoto

Someone make this into a Wiki entry so I can stop get notifications. Clearly not shithub issue material.

Believe me, I've thought about it. If it was a wiki page, it would still need an issue to be able to pin to the issues page. If locked, it would lead to people filing new issues. You can unsubscribe from this one more easily than unsubscribing from newly filed ones.

Jun 04 '23 01:06 philipl

Believe me, I've thought about it. If it was a wiki page, it would still need an issue to be able to pin to the issues page. If locked, it would lead to people filing new issues. You can unsubscribe from this one more easily than unsubscribing from newly filed ones.

I believe you, which is really sad. These web-based git front-ends have nothing new to offer nowadays in terms of usability, every shithub new "feature" for the past 4yrs or so is nothing more than a detriment for browser performance and with questionable interface changes... but I digress. Will unsubscribe from this issue and everyone can carry on.

Jun 04 '23 01:06 garoto

I usually use vo=dmabuf-wayland but noticed that it's causing a segfault when vulkan interop is compiled in (even when it's not being used). Seems like when using dmabuf-wayland it attempts to run through loading every hwdec option even if one is explicitly specified.

Happy to use gpu-next but thought I should note this bug. But also curious if this will work with dmabuf-wayland in the future?

It's a bug, and I've pushed a fix. Thanks for reporting.

Jun 04 '23 01:06 philipl

The Sony swordsmith sample got fixed on radv with the following mesa PR: https://gitlab.freedesktop.org/airlied/mesa/-/commits/radv-video-fix-hevc-wip There was also a bug in ffmpeg, fixed with https://github.com/FFmpeg/FFmpeg/commit/65a1e8ee2c4a9c2f325e1921daab915689d9c9fd. Could anyone check if that fixed it on nvidia?

Jun 06 '23 12:06 cyanreg