[Bug][Regression in API 2.10]: Arrayed DX11 texture fails to share with OpenCL without MFXVideoVPP_ProcessFrameAsync

[nyanmisaka]

Which component impacted?

Video Processing

Is it regression? Good in old configuration?

Yes, it's good in old version

What happened?

1. Using FFmpeg with Intel Arc A380 GPU on Windows 10 22H2
2. oneVPL API version 2.10 (graphics driver 31.0.101.5333)
3. Run FFmpeg command: (FFmpeg MUST be patched with D3D11_RESOURCE_MISC_SHARED flag to allow sharing)

./ffmpeg.exe -init_hw_device d3d11va=dx:,vendor_id=0x8086 -init_hw_device opencl=ocl@dx `
 -hwaccel_device dx -filter_hw_device ocl `
 -hwaccel d3d11va -hwaccel_output_format d3d11 -threads 2 `
 -i "any_h264_video_clip.mp4" -an -sn `
 -vf "hwmap=derive_device=opencl:mode=read,format=opencl,hwdownload,format=nv12" `
 -c:v libx264 "green_video.mp4"

4. The output video is green (empty YUV data), which means D3D11 arrayed texture to OpenCL sharing has failed.
5. But if you import the D3D11VA decoded frame into QSV and insert a dummy scale_qsv filter to invoke MFXVideoVPP_ProcessFrameAsync, D3D11 -> OpenCL sharing works.
6. This issue also applies to the QSV decoder of FFmpeg, but some additional modifications need to be made in hwcontext_opencl.c to allow indirect sharing between AV_PIX_FMT_QSV/D3D11 and AV_PIX_FMT_OPENCL. If you need these patches I can provide them.

What's the usage scenario when you are seeing the problem?

Transcode for media delivery

What impacted?

The existing sharing between AV_PIX_FMT_D3D11 and AV_PIX_FMT_OPENCL in FFmpeg does not work anymore on Intel GPU, but it still works on AMD.

What I'm expecting is direct sharing between D3D11 and OpenCL, not via VPP Async to convert an arrayed texture to an intermediate texture and do sharing with it, which takes a performance hit since I've guaranteed synchronization of my provided D3D11 texture and there's a performance constraints of the fixed-function VPP hardware.

I previously reported the issue in Intel-GPU-Community-Issue-Tracker, and it seems that the developers there cannot locate the issue. https://github.com/IGCIT/Intel-GPU-Community-Issue-Tracker-IGCIT/issues/680

Debug Information

• This is a regression because in graphics driver 31.0.101.5085 and earlier it works fine, and the oneVPL API for these drivers is 2.9, which means they do not contain the VPL Surface Sharing mechanism newly introduced in 2.10.

• The cl_intel_d3d11_nv12_media_sharing OpenCL extension provide support for sharing arrayed D3D11 texture with OpenCL. (D3D11_TEXTURE2D_DESC.ArraySize > 1)

• Sharing Surfaces between OpenCL™ and DirectX* 11 on Intel® Processor Graphics

Do you want to contribute a patch to fix the issue?

No.

[nyanmisaka]

cc @mgonchar, who is the author of commit d2470d2

Surface Sharing: Initial implementation (#5708)
- Basic framework
- Sharing of native handles (VASurfaceId on Linux)
- OCL sharing on Win (copy / no copy Export; only copy path for Import)
- Capability reporting

[nyanmisaka]

@FurongZhang Sorry to trouble you again. This issue is not specific to use VPP hardware but is related to do VPP in OpenCL kernel. Could you please let intel developers who is familiar with QSV/DXVA and OpenCL aware of this issue? Thanks in advance.

[FurongZhang]

@jonrecker , @xhaihao , any suggestions here?

[xhaihao]

@nyanmisaka It sounds to me that this is a regression in OpenCL, not VPL (You should be able to build FFmpeg without VPL support, then run your command with the old driver, hence VPL is not required for your command, the regression is irrelative to VPL).

[nyanmisaka]

@xhaihao Thanks for your time. I mean the underlying D3D runtime is affected by this issue. So the VPL runtime is also affected since it's inherited from D3D/DXVA. But I can't be sure which side of the D3D or OpenCL runtime the problem is, because unlike on Linux, these runtime are all closed source on Windows so I can't compile or bisect. I did search in compute-runtime/NEO for recent changes about D3D11, but didn't find anything significant.

Any chance you guys can reproduce the issue and open an issue ticket for this internally?

[xhaihao]

@nyanmisaka We can reproduce your issue and will file an internal issue.

[tong1wu]

@nyanmisaka We have root caused the specific commit and filed an issue internally. Thanks for reporting.

[nyanmisaka]

@xhaihao @tong1wu Glad to know this and thanks for your help. Looking forward to re-testing in the future driver.

[yuqinw]

hi, in the 3rd step: _Run FFmpeg command: (FFmpeg MUST be [patched with D3D11_RESOURCE_MISC_SHARED]） I download official released ffmpeg.exe find the source code don't have the change. And I can not build a passed ffmpeg.exe. So could someone can help provide me a useful ffmpeg bin? Thanks for your support in advance.

[nyanmisaka]

@yuqinw Our project uses a customized ffmpeg which includes this patch, so it can be used to reproduce this issue.

[hye5]

Fix in gfx-driver-ci-comp_vulkan-9789

[nyanmisaka]

@hye5 Thank you! May I know how long it will take for this to be publicly available as a new driver release?

[nyanmisaka]

Fixed in 31.0.101.5592. Thank you everyone!