MediaSDK
MediaSDK copied to clipboard
Intel-Media-SDK
Fill HDR Side Data for HEVC Decoding
Structures for HDR side data have recently been added for HEVC encoding: https://github.com/Intel-Media-SDK/MediaSDK/commit/a9685bf658d1ec97ff44b614dcf75818ecc62819 :
- mfxExtContentLightLevelInfo
- mfxExtMasteringDisplayColourVolume
What missing is to fill those structures when decoding HEVC video.
Could this be added?
@softworkz , just to be on the same page: would you like to have this reporting from DecodeHeader and together with each output frame (if there're per frame SEI), right?
@softworkz , just to be on the same page: would you like to have this reporting from DecodeHeader and together with each output frame (if there're per frame SEI), right?
Correct. From all videos that I've seen so far, even with static HDR (like HDR10), the data is in every frame (SEI type 137 and 144 IIRC).
Purposes:
- Re-Encoding while keeping HDR data e.g. just scaling down while retaining color format and HDR (you've already done the encoding side)
- The data is required for tone mapping
- It's a prerequisite for this: https://github.com/Intel-Media-SDK/MediaSDK/issues/1949
- But even right now it's a problem when doing tone mapping for example with OpenCL, because QuickSync decoders (unlike VAAPI) do bitstream and codec parsing, so it's not possible to get at this data (e.g. using ffmpeg)
Thanks for details. Yes, we can do it but please expect some delay, ~2 weeks.
Thanks for details. Yes, we can do it but please expect some delay, ~2 weeks.
@dmitryermilov @pengxin99 - Is there any progress in sight on this issue?
@softworkz Sorry for slow response. As you know, we are transferring from MSDK legacy to oneVPL (https://github.com/oneapi-src/oneVPL-intel-gpu). For legacy branch, it has been implemented by @pengxin99 with extension buffer. But this flow cannot be used in VPL architecture. We are evaluating the path and will update it ASAP.
Only the "legacy" path is of interest at this time. Will the implementation from @pengxin99 be merged?
For Linux we could easily merge this patch ourselves, but not for Windows - that's the big problem. Or is there a way how we could build our own libmfxhw64.dll for Windows?
@softworkz In general, we will not add new feature into legacy path and just bug fix. Of course, you could use @pengxin99's legacy PR. Windows code isn't included in current repo (https://github.com/Intel-Media-SDK). @dmitryermilov @onabiull Could we provide binary build beyond the normal release build?
Or even better - could we get the Windows code (maybe under NDA) to build our own binaries? Thanks, sw
Binary for NDA user - I assume it's possible. Code for NDA user - there should be a very strong request, unlikely to be approved but I can double check. Please send me a formal email including company name.
Fixed by https://github.com/Intel-Media-SDK/MediaSDK/pull/2726. Manual update: https://github.com/Intel-Media-SDK/MediaSDK/pull/2876.
Windows changes will be merged today. Please expect a few (2-8) weeks delay to get updated msdk/vpl run-time library in public graphics driver packages.
Hey Dmitry,
thanks that's great news!
Now we will just need to let it land in ffmpeg ;-)
I'll check with @xhaihao whether he wants to do it. Otherwise I'll try to find some time.
Thanks again, softworkz
@softworkz Yes, we will check the latest driver. @dmitryermilov it is available in the latest oneVPL GPU runtime on both Linux & Windows, right ? Does MediaSDK support it ?
@xhaihao Yes, It is common code for both Windows and Linux. MediaSDK also supports it.
@dmitryermilov @wangyan-intel If user provides MFX_EXTBUFF_MASTERING_DISPLAY_COLOUR_VOLUME & MFX_EXTBUFF_CONTENT_LIGHT_LEVEL_INFO extended buffers, is there a flag indicating valid HDR data is returned from the SDK?
Good question, @xhaihao . I have to admit it looks as a gap. I can propose only to rely on checking non zero MaxDisplayMasteringLuminance/MaxContentLightLevel ..
however 0 is a valid value for max_content_light_level and max_pic_average_light_level
@dmitryermilov @wangyan-intel If user provides MFX_EXTBUFF_MASTERING_DISPLAY_COLOUR_VOLUME & MFX_EXTBUFF_CONTENT_LIGHT_LEVEL_INFO extended buffers, is there a flag indicating valid HDR data is returned from the SDK?
@pengxin99 Could you please clarify it? Thanks
Are RGB x coordinates stored in DisplayPrimariesX[3] in R=>G=>B order, or G=>B=>R order ?
hi, @xhaihao , At first, we wanted to use InsertPayloadToggle in the struct mfxExtMasteringDisplayColourVolume to indicate whether the decoded clips have HDR SEI, but the InsertPayloadToggle was designed for encoding originally, so we did not use this plan.
The current way is that if the value of all members in the struct mfxExtMasteringDisplayColourVolume are 0, it means that there is no HDR sei in the decoded clips, and vice versa.
is it possible to use a reserved byte in mfxExtMasteringDisplayColourVolume and mfxExtContentLightLevelInfo to indicate whether the SDK writes valid data back ?
Are RGB x coordinates stored in DisplayPrimariesX[3] in R=>G=>B order, or G=>B=>R order ?
For the DisplayPrimariesX[3], We just parse the SEI of clips storage information in order, and according to the spec, it should be G=>B=>R order.
is it possible to use a reserved byte in
mfxExtMasteringDisplayColourVolumeandmfxExtContentLightLevelInfoto indicate whether the SDK writes valid data back ?
I think this way can be considered, I will discuss it with @wangyan-intel @dmitryermilov for this. If it is fine, I will continue to add this flag.