VA_EXPORT_SURFACE_COMPOSED_LAYERS not supported in vaExportSurfaceHandle that required by mpv dmabuf-wayland output

[klozher]

vaExportSurfaceHandle does not support VA_EXPORT_SURFACE_COMPOSED_LAYERS, actually, it returns VA_STATUS_ERROR_INVALID_SURFACE directly if VA_EXPORT_SURFACE_SEPARATE_LAYERS not present in flags

https://github.com/elFarto/nvidia-vaapi-driver/blob/c519e97ef7af581c109f49b6973269fb16d1bc54/src/vabackend.c#L2099-L2114

mpv with dmabuf-wayland output calls vaExportSurfaceHandle with VA_EXPORT_SURFACE_COMPOSED_LAYERS flag, thus failed to make any output.

[elFarto]

I'm not entirely sure we can support that as NVDEC can only export as separate layers. There's no reason to use this library with mpv, it can just use NVDEC directly.

[philipl]

The dmabuf-wayland vo in mpv cannot work with nvdec, as there's no way to do native dmabuf interop without doing the same tricks this driver does. But even if it did, we'd have the same problem with separate vs composed frames.

Unfortunately, it means dmabuf-wayland won't work on Nvidia. Mind you, I don't think any of the efficiency arguments for it carry much weight when using a Nvidia GPU.

[klozher]

i did some test before, by modifying direct_fillExportDescriptor of direct backend, something like

    for (uint32_t i = 0; i < fmtInfo->numPlanes; i++) {
        desc->layers[0].object_index[i] = 0;
        desc->layers[0].offset[i] = img->offsets[i];
        desc->layers[0].pitch[i] = img->strides[i];
    }

it seems work for me, mpv can play some video with dmabuf-wayland, though i don't know much about graphic stacks nor video stuff, i wonder is this change valid for this issue, is it really work or i was mistaken about something

[elFarto]

That won't work. The driver exports 2 FDs, each to one of the images, the Y (luma) and the UV (chroma) images. What you're doing is pointing the chroma to the luma image. That will work technically, but it won't look right.

I did try to export the luma and chroma in a single FD, but the NVIDIA driver has weird swizzling rules for images under a certain size, so you can get yourself into a situation where the luma and chroma needs different DRM modifiers to import them and I don't think I can work around that.