HDR screenshots are unimplemented (gpu/gpu-next)

[low-batt]

mpv Information

mpv v0.39.0-129-g4c24eae220 Copyright © 2000-2024 mpv/MPlayer/mplayer2 projects
 built on Oct 15 2024 20:49:12
libplacebo version: v7.349.0 (v7.349.0-18-g9e16c86f)
FFmpeg version: n7.1
FFmpeg library versions:
   libavcodec      61.19.100
   libavdevice     61.3.100
   libavfilter     10.4.100
   libavformat     61.7.100
   libavutil       59.39.100
   libswresample   5.3.100
   libswscale      8.3.100

Other Information

- macOS version: macOS 13.6.9 (22G830)
- Source of mpv: My own build of master using mpv-build
- Introduced in version: Unknown

Reproduction Steps

The version of IINA under development used to generate JPEG XL screenshots that used HDR color spaces. I recently tested it and found the screenshots looked muddy.

Running mpv on a Mac that supports EDR with --vo=libmpv and setting other options similar to what IINA does causes the video to be displayed with EDR enabled:

mpv/build/mpv --no-config --gpu-debug --log-file=output.txt --autofit=100% --cocoa-cb-output-csp=bt.2100-pq  --pause --screenshot-format=jxl --start=6 --target-prim=bt.2020 --target-trc=pq --vo=libmpv ~/Movies/The\ World\ in\ HDR\ in\ 4K\ \(ULTRA\ HD\).mp4

Take a screenshot. Use jxlinfo to inspect the resulting JPEG XL screenshot. Notice the screenshot uses sRGB.

Expected Behavior

A HDR screenshot would be generated.

Actual Behavior

The resulting JPEG XL screenshot:

low-batt@gag mpv-build (master %=)$ jxlinfo mpv-shot0001.jxl 
JPEG XL file format container (ISO/IEC 18181-2)
JPEG XL image, 3840x2160, lossy, 16-bit RGB+Alpha
Color space: RGB, D65, sRGB primaries, sRGB transfer function, rendering intent: Relative
low-batt@gag mpv-build (master %=)$ 

From log file:

[   6.457][v][cplayer] Starting screenshot: 'mpv-shot0001.jxl'
[   6.457][v][screenshot] input: 3840x2160 [0:0] rgba64 rgb/bt.709/srgb/full/display CL=mpeg1/jpeg
[   6.457][d][cplayer] Run command: script-binding, flags=73, args=[name="osc/__keybinding4"]
[   6.458][v][screenshot] will convert image to rgba64
[   6.458][v][screenshot] using 8 threads for scaling
[   6.461][v][screenshot] Using zimg.
[   6.475][d][screenshot] mapped color params:
[   6.475][d][screenshot]   trc = iec61966-2-1
[   6.475][d][screenshot]   primaries = bt709
[   6.475][d][screenshot]   range = pc
[   6.475][d][screenshot]   colorspace = gbr
[   6.475][d][screenshot]   chroma_location = center
[   7.137][i][cplayer] Screenshot: 'mpv-shot0001.jxl'

Adding --screenshot-sw=yes and repeating the test (looks better than this when viewed with a JPEG XL viewer):

low-batt@gag mpv-build (master %=)$ jxlinfo mpv-shot0002.jxl 
JPEG XL image, 3840x2160, lossy, 16-bit RGB
intensity_target: 10000.000000 nits
min_nits: 0.000000
relative_to_max_display: 0
linear_below: 0.000000
Color space: RGB, D65, Rec.2100 primaries, PQ transfer function, rendering intent: Relative
low-batt@gag mpv-build (master %=)$ 

For completeness, with gpu-next:

mpv/build/mpv --no-config --autofit=100% --pause --screenshot-format=jxl --start=6 --target-colorspace-hint --vo=gpu-next ~/Movies/The\ World\ in\ HDR\ in\ 4K\ \(ULTRA\ HD\).mp4

EDR is activated and the resulting screenshot is similar to libmpv with screenshot-sw, but slightly different:

low-batt@gag mpv-build (master %=)$ jxlinfo mpv-shot0003.jxl 
JPEG XL file format container (ISO/IEC 18181-2)
JPEG XL image, 3840x2160, lossy, 16-bit RGB+Alpha
intensity_target: 10000.000000 nits
min_nits: 0.000000
relative_to_max_display: 0
linear_below: 0.000000
Color space: RGB, D65, Rec.2100 primaries, PQ transfer function, rendering intent: Relative

Log File

output.txt Screenshots.zip

Sample Files

I used a downloaded copy of this YouTube video: The World in HDR in 4K (ULTRA HD)

I carefully read all instruction and confirm that I did the following:

• [X] I tested with the latest mpv version to validate that the issue is not already fixed.
• [X] I provided all required information including system and mpv version.
• [X] I produced the log file with the exact same set of files, parameters, and conditions used in "Reproduction Steps", with the addition of --log-file=output.txt.
• [X] I produced the log file while the behaviors described in "Actual Behavior" were actively observed.
• [X] I attached the full, untruncated log file.
• [X] I attached the backtrace in the case of a crash.

[sfan5]

Is your intent to take a screenshot that is itself HDR or just a (tonemapped) screenshot of HDR video? I'm assuming the former.

I might be wrong but ~~storing an HDR image as RGB sounds fundamentally wrong to me~~. Since VO screenshots always return RGB that means this could possibly only work with --screenshot-sw --screenshot-tag-colorspace --screenshot-high-bit-depth and a format that takes YUV >=10-bits. ~~Note that ffmpeg's JXL support doesn't take YUV so is unsuited for this.~~ AVIF might work.

---

If you want to the latter: In case of --screenshot-sw=no the VO probably returns ~~either~~ mistagged RGB ~~or RGB but with a HDR csp~~. seems weird and broken?? You could argue that it's the VO's responsibility to produce a tonemapped screenshot even if the native display is HDR (how will this be configured?). In case of --screenshot-sw=yes we'd need to tonemap in software with sws or zimg. Problem is neither can. zimg produces the least bad but still incorrect result, ~~probably what you were seeing with "Adding --screenshot-sw=yes and repeating the test"~~.

[low-batt]

The intent is to obtain a HDR screenshot that matches what is seen on the screen, not a tone mapped screenshot. A JPEG XL screenshot that would activate EDR (with a viewer and a Mac that supports it). That is what is generated by --vo=gpu-next and what is generated with --vo=libmpv if I add --screenshot-sw=yes.

I believe at some point that is what mpv was generating without --screenshot-sw=yes as I don't remember needing to use that option when I was originally testing JPEG XL screenshot generation.

[haasn]

This is a vo_gpu limitation, no? It doesn't implement HDR screenshots. Shouldn't be too hard to support.

As a possible short term work around, we could fallback to sw screenshots if the VO does not support HDR screenshots.

[kasper93]

I have branch with refactored screenshot code in mpv to resolve some common issues. Though was too lazy to finish it. Maybe I will get back to it. In the meantime if someone wants to fix vo_gpu and others, don't hesitate.

[sfan5]

so to be clear the screenshot under

Adding --screenshot-sw=yes and repeating the test (looks better than this when viewed with a JPEG XL viewer):

is perfectly fine? and RGB with a HDR colorspace is no problem?

[haasn]

is perfectly fine? and RGB with a HDR colorspace is no problem?

JXL converts everything to XYZ anyway, doesn't it? So that's just one less conversion step.

[low-batt]

I played the movie with --vo=libmpv and --screenshot-sw=yes:

mpv/build/mpv --no-config --autofit=50% --cocoa-cb-output-csp=bt.2100-pq --script-opts-append=osc-visibility=always --pause --screenshot-format=jxl --screenshot-sw=yes --start=6 --target-prim=bt.2020 --target-trc=pq --vo=libmpv ~/Movies/The\ World\ in\ HDR\ in\ 4K\ \(ULTRA\ HD\).mp4

Took a screenshot and then displayed the screenshot:

mpv/build/mpv --no-config --autofit=50% --cocoa-cb-output-csp=bt.2100-pq --pause --script-opts-append=osc-visibility=always --target-prim=bt.2020 --target-trc=pq --vo=libmpv mpv-shot0001.jxl

I then took a picture of my MacBook Pro (XDR screen). The screenshot is on top. To my eyes the screenshot looks like the video: IMG_1070.HEIC.zip