HEVC Playback on Raspberry Pi 4

[EmperorPenguin18]

Playback of HEVC video can be slow on the RPI 4. It is hard to pin down what causes the slowness. I see it most often when there are complex subtitles and fast moving images like during anime intros, but it also happens in situations nothing like that. From a practical perspective, I understand the sentiment that this is a very uncommon use-case, and someone could just buy better / non-broadcom hardware. However from a technical perspective, I would consider this missing functionality. LibreELEC and VLC both have hardware acceleration working and while I'm aware they have a very different stack and it can't just be added to MPV, it does show that it is possible. I'm not sure if something has to happen in the Raspberry Pi kernel or ffmpeg first, but I thought I'd start here.

[CounterPillow]

mpv doesn't have to do anything special if Broadcom and their tax-evasion "raspberry pi foundation" scheme just followed standard hardware acceleration APIs.

[Traneptora]

As far as I'm aware, mpv already supports --hwdec=mmal, so what happens when you try that?

[EmperorPenguin18]

Adding that flag causes no noticeable change in behaviour for me. Setting it to auto gives an error related to v4l2m2m.

If what you're saying is that my experience is unexpected behaviour then I'll provide some more information.

Mpv version: 0.34.1-dirty Linux distribution and version: Manjaro aarch64 Raspberry Pi minimal Source of the mpv binary: Package manager Window manager and version: Sway 1.7 GPU driver and version: KMS? Not really sure how the Pi's stack works. Kernel is 5.15.74-2-MANJARO-ARM-RPI and mesa is 22.2.1-1

Logs and config file are attached below: default --hwdec=mmal --hwdec=auto config file

[nyanmisaka]

HEVC decoder on RPi4 requires some V4L2 changes in ffmpeg: https://github.com/jc-kynesim/rpi-ffmpeg/branches/active

[EmperorPenguin18]

So if I was to manually build ffmpeg from one of those branches and replace the system binary with mine, would HEVC decode on RPI4 just start working?

[CounterPillow]

no because mpv needs to be built against the correct ffmpeg binary, you cannot simply replace your system one.

Instead, build ffmpeg with a --prefix (some directory you create), make install it to install it into that prefix, then build mpv against it with PKG_CONFIG_PATH=/path/to/your/prefix/lib/pkgconfig/ meson build && ninja -C build

[philipl]

But to explicitly answer the question: if you have the right ffmpeg and build mpv against it, then yes, in theory mpv should work correctly with the hevc decoding, but no mpv dev has ever been in a position to test it, so YMMV.

[EmperorPenguin18]

I've tried a few different configurations now.

If I install this ffmpeg: ffmpeg-rpi and compile mpv with that as the pkg-config and set --hwdec=drm (or --hwdec=v4l2m2m for H264) then it plays but gives the error

[vo/gpu] Initializing texture for hardware decoding failed.

and just shows a blue screen instead of the video. Here is the log for that setup: arch.txt

Manually building mpv against that ffmpeg, the stock ffmpeg provided by Manjaro or this AUR package: ffmpeg-v4l2-request seem to give the same result; much worse playback of both HEVC and H264. I assume this is some configuration error on my part when building mpv, or maybe the mpv provided by Manjaro has something else going on. Here is the log for that third setup: aur.txt

Lastly, I tried building the ffmpeg linked above, specifically the dev/5.1.2/rpi_import_1 branch but because I'm a complete noob I couldn't get the make command to do anything. I can configure it using the provided conf_native.sh script (after changing the dpkg line) but running make gives the error that it can't find /tests/Makefile.

The first configuration seems promising. That's the closest I've gotten to hardware decoding with HEVC on the Pi. Could it be something wrong with my setup maybe? Something in /boot/config.txt or mpv.conf that causes that? Maybe there's something missing from that build of ffmpeg?

[EmperorPenguin18]

Looking for potential sources for the error, here is what I have found.

The actual error output happens at line 899 of video/out/gpu/video.c. It happens when ra_hwdec_mapper_create() fails. That function is defined starting at line 127 of video/out/gpu/hwdec.c. This function can only return NULL if mapper->driver->init(mapper) returns a value less than 0. I assume this function will be the one defined starting at line 162 in video/out/hwdec/hwdec_drmprime.c because of the hwdec type I'm trying to use (though I may be mistaken).

This function has multiple fail states but hopefully that helps narrow the problem down a little.

Also to confirm all the variables are the same, I'm running with --no-config and using this file: https://user-images.githubusercontent.com/60635017/204118697-7269689f-5fb8-4f7f-ae76-38045e27b355.mp4 (you may be able to tell that this isn't the file I originally got the logs with but it has the same result). I have also updated my system since my second comment so the version of stuff is different.

[philipl]

Your first log shows it appearing to work - the drmprime hwdec initialises and we see it trying to do stateless v4l2 decoding. There are no errors, so I don't know why it's just showing blue. Is there anywhere that this hevc decoder works? Your second log shows no evidence that the drmprime hwdec is present - we don't even see it try to initialise.

[EmperorPenguin18]

There is an error, it's that initializing failed line. The only change in behaviour I could get was if I re-encode the video I linked above with libx264rgb then try to play it with hwdec then it fails to load at all. rgb.txt

[EmperorPenguin18]

I also added printfs to the different return -1;s in mapper_init and with HEVC files it fails with the first one:

if (mapper->ra->num_formats &&
        !ra_get_imgfmt_desc(mapper->ra, mapper->dst_params.imgfmt, &desc))
    return -1;

and with H264 files it fails with the third one (also identifiable because of the unsupported DRM image format output):

if (!check_fmt(mapper, mapper->dst_params.imgfmt))
{
    MP_FATAL(mapper, "unsupported DRM image format %s\n",
             mp_imgfmt_to_name(mapper->dst_params.imgfmt));
    return -1;
}

I also get the third one if I comment out the first one and play HEVC files.

For that first check, num_formats is always set to 27, so that isn't the culprit. It seems it is the ra_get_imgfmt_desc function, which is defined starting at line 276 of video/out/gpu/ra.c. This function returning false seems to be related to the unsupported format error.

[EmperorPenguin18]

I noticed that when hwdec=no the output format is yuv420p, but with hwdec=auto it is rpi4_8. That text is nowhere in the code base so it must come from a library. On my laptop which uses vaapi, instead of rpi4_8 it becomes nv12. This image format concept is something I know nothing about, but seems to maybe be the core of the issue. This page helps me understand it better. It seems the video file is stored on disk in yuv420p format, but the hardware decoder accepts nv12, so a conversion needs to be performed. If the Raspberry Pi works a similar way and wants rpi4_8 format, maybe that is where the issue is.

[EmperorPenguin18]

A quick search brings up these results https://forum.kodi.tv/showthread.php?tid=367348 https://forums.raspberrypi.com/viewtopic.php?t=320638 https://github.com/LibreELEC/LibreELEC.tv/blob/master/packages/multimedia/ffmpeg/patches/v4l2-drmprime/ffmpeg-001-v4l2-drmprime.patch Which seem to be related to the yuv420p to drm_prime conversion.

[Traneptora]

Those are just hardware-specific pixel formats. No conversion is done before decoding as that would not be possible to do until after decoding. When decoding directly to a hardware texture, without using the copy-back decoders, those are used instead.

[EmperorPenguin18]

Those are just hardware-specific pixel formats. No conversion is done before decoding as that would not be possible to do until after decoding. When decoding directly to a hardware texture, without using the copy-back decoders, those are used instead.

Than what causes the unsupported image format error from this function?

I also added printfs to the different return -1;s in mapper_init and with HEVC files it fails with the first one:

if (mapper->ra->num_formats &&
        !ra_get_imgfmt_desc(mapper->ra, mapper->dst_params.imgfmt, &desc))
    return -1;

and with H264 files it fails with the third one (also identifiable because of the unsupported DRM image format output):

if (!check_fmt(mapper, mapper->dst_params.imgfmt))
{
    MP_FATAL(mapper, "unsupported DRM image format %s\n",
             mp_imgfmt_to_name(mapper->dst_params.imgfmt));
    return -1;
}

I also get the third one if I comment out the first one and play HEVC files.

For that first check, num_formats is always set to 27, so that isn't the culprit. It seems it is the ra_get_imgfmt_desc function, which is defined starting at line 276 of video/out/gpu/ra.c. This function returning false seems to be related to the unsupported format error.

[philipl]

Ok. i found the relevant error in the original log:

[   0.441][i][cplayer] VO: [gpu] 1920x1080 drm_prime[rpi4_8]
[   0.442][v][cplayer] VO: Description: Shader-based GPU Renderer
[   0.442][v][vo/gpu] reconfig to 1920x1080 drm_prime[rpi4_8] bt.709/bt.709/bt.1886/limited/display SP=1.000000 CL=mpeg2/4/h264
[   0.442][v][vo/gpu/wayland] Reconfiguring!
[   0.442][d][vo/gpu] max content size: 1920x1080
[   0.442][d][vo/gpu] monitor size: 1920x1080
[   0.442][v][vo/gpu] Resize: 1920x1080
[   0.442][v][vo/gpu] Window size: 1920x1080 (Borders: l=0 t=0 r=0 b=0)
[   0.442][v][vo/gpu] Video source: 1920x1080 (1:1)
[   0.442][v][vo/gpu] Video display: (0, 0) 1920x1080 -> (0, 0) 1920x1080
[   0.442][v][vo/gpu] Video scale: 1.000000/1.000000
[   0.442][v][vo/gpu] OSD borders: l=0 t=0 r=0 b=0
[   0.442][v][vo/gpu] Video borders: l=0 t=0 r=0 b=0
[   0.442][e][vo/gpu] Initializing texture for hardware decoding failed.

So yes, this is no surprise. Firstly, the rpi4_8 drm format isn't recognised by upstream ffmpeg (hence no grep match), but is presumably added by the rpi fork. And then, as it's a fruitcake format, there is obviously no possible GL or Vulkan texture format that could possibly match it for direct display, so the texture initialisation fails, and you get blue.

Conclusion: It's not going to work. It needs to produce a sane format like nv12 for mpv to work. Someone (not me) would need to write a sampler shader to do the conversion on the GPU as well as add a new code path to get the frame into GL/Vulkan as a buffer rather than a texture, etc, etc.

[EmperorPenguin18]

Awesome thanks for the informative reply. I'm going to do some research and see what I can come up with. I understand it is probably annoying working with nonsense non-standard hardware but I really appreciate everyone who has replied so far.

[EmperorPenguin18]

Here is all the different behaviour I've been able to get:

HEVC 8-bit https://user-images.githubusercontent.com/60635017/206880425-d81f4f23-b4d7-41b0-94e1-f3b8718d5d1a.mp4 hevc_8.txt Blue screen due to failed texture initialization.

HEVC 10-bit https://user-images.githubusercontent.com/60635017/206880440-27fee907-75d6-4c36-b230-e3bf57a2d7d5.mp4 hevc_10.txt Blue screen due to failed texture initialization.

H264 https://user-images.githubusercontent.com/60635017/206880471-4522299d-0078-4e14-9587-188891b731aa.mp4 h264.txt Blue screen due to failed texture initialization. This was just fixed by #10989

H264 4k (was too big so shortened) https://user-images.githubusercontent.com/60635017/206880576-4ff48a3b-4a68-4df8-a113-638a0eefa695.mov h264_4k.txt Falls back to software decoding because I believe the Pi doesn't support resolutions this high with H264.

H264 RGB https://user-images.githubusercontent.com/60635017/206880619-5659e7eb-57d2-40dd-b30d-e2518b4421df.mp4 h264_rgb.txt Doesn't load at all.

First file with --gpu-api=vulkan vulkan.txt Falls back to wlshm video output.

First file with --gpu-api=vulkan and mesa-git vulkan_git.txt Identical behaviour as without Vulkan.

In theory NV12 should be able to work but I haven't been able to get the Pi to output in that format, even though v4l2-ctl says it can.

When I have more time again I'll look into HEVC 8-bit because according to https://github.com/jc-kynesim/rpi-ffmpeg/issues/61 it should be possible, but will be more difficult than it was to get H264 working.

[EmperorPenguin18]

I've discovered another format: H264 10-bit https://user-images.githubusercontent.com/60635017/210462286-64c816be-7e1b-4187-9742-bbb46ee97abb.mp4 h264_10.txt Hardware decoding does work, however has poor performance and very noticeable artifacting. May need a separate issue to track this.

After trying some stuff I've found that there needs to be some changes in mesa to get it working, so I've created an issue here https://gitlab.freedesktop.org/mesa/mesa/-/issues/7944.

I have a branch that has working HEVC 8 and 10 bit videos here https://github.com/EmperorPenguin18/mpv/tree/pi_h265. The problem is it won't compile without the rpi-ffmpeg headers. How do I handle this? Add something to the meson build system? I can ask the question on #mpv-devel if it is more appropriate.

[EmperorPenguin18]

Anyone finding this later, read the linked pull request if you're curious about the details, but to get this working you need Mesa 23.1 which as of writing is set to release May 5th, and the rpi-ffmpeg fork, which you get differently depending on what distro you use. If you have troubles, please don't bother the nice mpv folks who have nothing to do with Pi problems, and just bother me instead. Speaking of, thank you to all mpv devs who helped me get this working! It's probably a meaningless fix for most people, but it has improved my video playing experience tremendously!

[vanfanel]

@EmperorPenguin18 Thanks for all the work you put into this! mpv + ffmpeg also has an special meaning to me, and your work is very much appreciated here :)

I am again trying to play H264 files with: -mpv latest GIT sources. -rpi-ffmpeg, branch dev/6.0/rpi_import_1, but I have tried all the active ones by now. -mesa-23.1.0-rc3

And what I see is that mpv can now play H264 using v4l2m2m, no more unsupported yuv240p format errors. Good!

However, using v4l2m2m causes constant frame drops (as in 24 frames dropped in 10 seconds!), while using software everything is fine (except for the very high CPU usage...)

Do you know what could be going on? Since I use mpv on Wayland, maybe this is expected to happen on Wayland?

[EmperorPenguin18]

Hi sorry for the delayed response, you caught me in the middle of moving and starting a new job.

I tried your described setup, but I couldn't reproduce your frame drops. Could you provide build options? Mpv config options? Maybe some details on the video you're testing with? I've stumbled on all sorts of problems in the past so it really could be any one of these or even something else.

[vanfanel]

@EmperorPenguin18 Of course.

I built rpi-ffmpeg from this branch: git clone --depth 1 https://github.com/jc-kynesim/rpi-ffmpeg.git -b dev/6.0/rpi_import_1

Configured rpi-ffmpeg like this:

CFLAGS="-march=native -mtune=native" CXXFLAGS="-march=native -mtune=native" ./configure --disable-network --disable-debug \
--disable-muxers --disable-indevs \
--disable-outdev=fbdev,oss \
--disable-doc --disable-bsfs --disable-ffprobe \
--disable-sdl2 --disable-stripping \
--disable-thumb --disable-mmal \
--enable-sand --enable-v4l2-request \
--enable-libdrm --enable-epoxy \
--enable-libudev --enable-vout-drm

Then I used this repo for mpv sources: git clone --depth 1 https://github.com/mpv-player/mpv.git

Configured MPV like this:

CFLAGS="-march=native -mtune=native" \
CXXFLAGS="-march=native -mtune=native" \
meson -Dbuildtype=release ..

And this is my ~/.config/mpv/mpv.conf file:

hwdec=auto
fs=yes
sub-visibility=no

By setting hwdec=auto the v4l2m2m coded is used, which apparently works fine.

BUT if I set video-sync=display-resample or video-sync=display-desync, frame dops happen. Note that video-sync=display-resample or `video-sync=display-desync are needed for actual smooth video playback.

Disabling hwdec=auto, there are no frame drops with video-sync=display-resample or video-sync=display-desync.

So I have to chose one or the other.... hwdec or video-sync, but they don't work well together :(

Video files are all h264, for example this one:

root@raspberrypi:~# mpv /media/BACKUP2/Peliculas2/Castaway\ on\ the\ Moon/noo\ Meht\ no\ yawatsacv.mp4 
 (+) Video --vid=1 (*) (h264 1920x816 23.976fps)
 (+) Audio --aid=1 --alang=kor (*) (aac 2ch 44100Hz)
File tags:
 Title: Castaway.on.the.Moon.2009.BluRay.1080p.5.1CH.x264-SmallAndHD
AO: [alsa] 44100Hz stereo 2ch float
VO: [gpu] 1920x816 yuv420p

[EmperorPenguin18]

I tried video-sync and I'm seeing the same frame drops. I didn't see it on a video with no audio nor a H265 video. I also wanted to try X but mpv wouldn't even start. I think I agree with the comment here https://github.com/mpv-player/mpv/issues/11647 that it is probably a hardware limitation of the Pi. The thing is, I'm not really sure what this option is supposed to do because the video playback is visually identical with or without video-sync.

[vanfanel]

I tried video-sync and I'm seeing the same frame drops. I didn't see it on a video with no audio nor a H265 video. I also wanted to try X but mpv wouldn't even start. I think I agree with the comment here #11647 that it is probably a hardware limitation of the Pi. The thing is, I'm not really sure what this option is supposed to do because the video playback is visually identical with or without video-sync.

Thanks for taking a look, really! :)

The sync option improves the video smoothness by synchronizing video rate to the physical screen refresh rate and resampling the audio accordingly. That's not a noticeably CPU-intensive operation in RetroArch (which does the same with the emulated systems refresh rates and audio) but seems to be too much in MPV for some reason.

Image quality is the same with or without it, but frame "pacing" is different.

[EmperorPenguin18]

https://github.com/mpv-player/mpv/wiki/Display-synchronization https://github.com/mpv-player/mpv/wiki/Interpolation https://github.com/haasn/interpolation-samples This stuff is pretty complicated, so I would confirm that RetroArch is doing the feature the exact same before comparing. I'm not sure which would be more intensive, HD video playback or emulation, but I have a feeling that it's a Pi issue nonetheless. Is accelerated audio a thing? Maybe that could help?

[vanfanel]

@EmperorPenguin18

The problem only happens with v4l2m2m. With v4l2m2m-copy, however, no frames are dropped. (In both cases, using video-sync=display-resample). Please try them both v4l2m2m and v4l2m2m-copy with video-sync=display-resample and you will see the difference.

So there is a problem here: it's not about CPU/memory bandwidth, there's something wrong with v4l2m2m that doesn't affect v4l2m2m-copy, which is VERY strange.

I opened a new issue here: https://github.com/mpv-player/mpv/issues/11707

[wyup]

Anyone finding this later, read the linked pull request if you're curious about the details, but to get this working you need Mesa 23.1 which as of writing is set to release May 5th, and the rpi-ffmpeg fork, which you get differently depending on what distro you use. If you have troubles, please don't bother the nice mpv folks who have nothing to do with Pi problems, and just bother me instead. Speaking of, thank you to all mpv devs who helped me get this working! It's probably a meaningless fix for most people, but it has improved my video playing experience tremendously!

Good job! I'm new to RPI4, but would like to get hevc acceleration on Raspi OS on mpv and get its sdr tone-mapping. Has Mesa 23.1 been merged into may 3rd 6.1.21 kernel Bullseye?

Is rpi-ffmpeg and mpv manual building necessary?

[EmperorPenguin18]

Has Mesa 23.1 been merged into may 3rd 6.1.21 kernel Bullseye?

My understanding is that mesa is fully user-space, and no kernel changes are required. Mesa 23.1 did release, so now it is just depending on if your distro is shipping it. If you are also new to linux and don't know how to check that, running apt list | grep libgles2-mesa in the terminal will show the version of mesa.

Is rpi-ffmpeg and mpv manual building necessary?

For now, yes. I think somewhere in this thread is compilation instructions.