mpv
mpv copied to clipboard
mpv-player
hwdec/vaapi: improve probing of supported sw formats
The logic around trying to establish what formats are supported by vaapi is confusing, and it turns out that we've been doing it wrong.
Up until now, we've been going through the list of decoding profile endpoints and checking the formats declared in those profiles to build our list.
However, the set of formats that the vaapi driver can actually support is potentially a superset of those supported by any given profile endpoint. This master list is exposed by libavutil via the av_hwframe_transfer_get_formats() function, and so we should use that list as our starting point.
Perhaps surprisingly, this change actually removes no code, because we still want the logic that enumerates endpoints and finds supported formats for endpoints. We need this because we must have at least one known sw format to initialise the hwframe_ctx with.
Additionally, while in the general case, av_hwframe_transfer_get_formats can return different formats depending on what format you initialise the hwframe_ctx with, I happen to have read the libavutil code, and it doesn't care, so we just need to call it once, we know we'll get back all known formats.
In practice, with an Intel vaapi driver, this will result in us supporting a handful of extra formats for hwuploads - particularly yuv420p (so no need to convert to nv12 first) and various orderings of rgb(a).
Added a second change that gets rid of the "shit list" in hwtransfer, which was hard-coding supported formats in vaapi. It's not required anymore.
The code changes seem to be fine in general but I found some issues. mpv detects the following fmts:
[vo/gpu/vaapi] Going to probe surface formats (may log bogus errors)... [vo/gpu/vaapi] Failed to create mapper [vo/gpu/vaapi] Unsupported format: yuyv422 [vo/gpu/vaapi] Failed to create mapper [vo/gpu/vaapi] Unsupported format: uyvy422 [vo/gpu/vaapi] Supported formats: [vo/gpu/vaapi] nv12 [vo/gpu/vaapi] p010 [vo/gpu/vaapi] yuv420p [vo/gpu/vaapi] bgra [vo/gpu/vaapi] rgba [vo/gpu/vaapi] bgr0 [vo/gpu/vaapi] rgb0
Testing with ./build/mpv --no-config https://0x0.st/ovun.gif --loop-file=inf -vf format=$FMT,format=vaapi, the colors are wrong for FMT = bgr0, bgra or yuv420p (?!).
We dug a bit and found that the bad colours are specific to OpenGL. Vulkan works fine.
[vo/gpu-next/vaapi] Supported formats:
[vo/gpu-next/vaapi] nv12
[vo/gpu-next/vaapi] bgra
[vo/gpu-next/vaapi] argb
[vo/gpu-next/vaapi] rgba
[vo/gpu-next/vaapi] abgr
[vo/gpu-next/vaapi] bgr0
[vo/gpu-next/vaapi] 0rgb
[vo/gpu-next/vaapi] rgb0
[vo/gpu-next/vaapi] 0bgr
[vo/gpu-next/vaapi] vuyx
[vo/gpu-next/vaapi] yuv420p
[vo/gpu-next/vaapi] p010
[vo/gpu-next/vaapi] p012
[vo/gpu-next/vaapi] xv36
It looks like there's a couple of problems.
All the rgb[a|0] family of formats map to rgba8 in OpenGL, so the dmabuf import with eglCreateImageKHR appears to do swizzling to fix the ordering. But we look at the imgfmt to decide channel order, so if the import implicitly fixes the order, and then our code tries to fix it too, so we get a mess.
On top of that, there is something weird going on with the pixfmt to fourcc mappings in ffmpeg. I see that even though there is a unique fourcc for each flavour, they are getting mapped to the same fourccs in some cases, so it's possible we have the wrong fourcc, leading to the wrong swizzling in the import and then poor us trying to use the channel order from the original pixfmt.
I've added a work-around for the double swizzling for 4/6 cases. The other two have additional problems coming from somewhere else, currently unknown. I think this is good enough that we can ignore it. 0rgb and 0bgr as extra obscure and only exposed on the lasted Intel hardware anyway.
Adding a comment about the problems with 0rgb and 0bgr for future reference would be a good idea.
