jellyfin-ffmpeg
jellyfin-ffmpeg copied to clipboard
jellyfin
Feature request: enable/integrate rockchip hwaccel support
It would be really nice if the jellyfin-ffmpeg would support the rockchip hardware acceleration.
For 4.1 there's a maintained ffmpeg-fork here with additional patches it seems: https://github.com/rockchip-linux/ffmpeg/tree/rockchip/release/4.1 but basic rockchip acceleration support was already included via the RK-MPP in ffmpeg-3.
This library is needed to enable support: https://github.com/rockchip-linux/mpp
Communication with the kernel is then through a /dev/videoN device if I understood it correctly.
We'd like to keep in line with upstream ffmpeg as much as possible. Do you know if ffmpeg 4.2 or newer has this enabled?
It seems that support for rockchip-acceleration was added in ffmpeg-3 (and continues to be there). It is by default not enabled since it needs the mpp library:
~/Downloads/ffmpeg-4.2.2 $ ./configure --help | grep mpp
--enable-rkmpp enable Rockchip Media Process Platform code [no]
Looking at their repo I don't see a license. We cannot redistribute builds without compatible licenses... please investigate the licensing of their code and report back :)
The debian control file and the headers say Apache 2.0 (which is probably why ffmpeg requires to be built with --enable-version3):
- https://github.com/rockchip-linux/mpp/blob/develop/debian/copyright
- https://github.com/rockchip-linux/mpp/blob/develop/mpp/inc/mpp.h
Just did a succesful buidl test with the MPP library from their devel branch (the latest release is quiet old) and ffmpeg with ./configure --enable-rkmpp --enable-version3 --enable-libdrm.
I think you can try and make a PR yourself, this is probably the file you need to modify: https://github.com/jellyfin/jellyfin-ffmpeg/blob/jellyfin/docker-build.sh
I am also interested in having rockchip hwaccel, specifically for the Helios64. I've noticed that the mpp repo https://github.com/rockchip-linux/mpp doesn't exist anymore. There is a alternative here https://github.com/HermanChen/mpp. Don't know how official it is.
Is this a problem for redistribution?
I am also interested in having rockchip hwaccel, specifically for the Helios64. I've noticed that the mpp repo
https://github.com/rockchip-linux/mppdoesn't exist anymore. There is a alternative here https://github.com/HermanChen/mpp. Don't know how official it is.Is this a problem for redistribution?
I think it's still official. It has been moved to a personal repo of a Rockchip employee. There are only decoders for rkmpp in ffmpeg, encoders and scaler have not been implemented.
Any news on this one ? I am also very insterested in having hwaccel enabled for jellyfin upstream.
Does anyone know what actually needs to be done to enable this?
If I understand correctly:
- [ ] Package rockchip-mpp in the Jellyfin repo
- [ ] Add it to the requirements arm64 build and enable the build flag
You are free to do it. But note that ffmpeg only has rockchip’s hardware decoder, which means transcoding speed will still bottlenecked by software scaling and software encoding.
@nyanmisaka absolutely. That is why I personnaly moved to x86 platform whith much much better performance.
You are free to do it. But note that ffmpeg only has rockchip’s hardware decoder, which means transcoding speed will still bottlenecked by software scaling and software encoding.
That makes sense, I think this will have a reasonable benefit on rockchip servers.
Does my todo list make sense? If so, how would I get started with packaging rockchip-mpp in the jellyfin repo?
Sure, make sense. You can add related build configs in: https://github.com/jellyfin/jellyfin-ffmpeg/blob/jellyfin/debian/rules https://github.com/jellyfin/jellyfin-ffmpeg/blob/jellyfin/docker-build.sh
At least, the Armbian edge kernel for helios64 has hantro-vpu encoder/decoder and rockchip_vdec support. FFmpeg support is not official but active upstream seems to currently be https://github.com/jernejsk/FFmpeg (devel in vc1-v2 branch) shipped by LibreELEC in https://github.com/LibreELEC/LibreELEC.tv/tree/master/packages/multimedia/ffmpeg/patches/v4l2-request. Maybe https://github.com/LibreELEC/LibreELEC.tv/blob/master/packages/multimedia/ffmpeg/patches/v4l2-drmprime/ffmpeg-001-v4l2-drmprime.patch is of use too, but I do not know if required. I have not yet tested the above, but at first glance, a few patches are not required for non-Kodi FFmpeg. I have an helios64 with the Jellyfin docker container so I will be eager to test.
Another option might be the vaapi v4l2-request backend https://github.com/bootlin/libva-v4l2-request (this repository is inactive but there are newer low activity forks)
I think this is a good idea because it moves away from the typical: "Oh you need an intel/amd processor with integrated graphics to support transcoding" statement. And its another thing which makes jellyfin different from the rest (emby, plex).
The --enable-rkmpp flag exists in the configure script, but is defaulted to off.
I mainly use the Jellyfin Docker container. Any good way that I can replace that with a self-built jellyfin-ffmpeg that has rkmpp enabled - and then actually use it in Jellyfin itself?
The
--enable-rkmppflag exists in the configure script, but is defaulted to off.I mainly use the Jellyfin Docker container. Any good way that I can replace that with a self-built jellyfin-ffmpeg that has rkmpp enabled - and then actually use it in Jellyfin itself?
The fact is that rkmpp hwaccel in ffmpeg is outdated to support rk35xx chips, and it doesn’t support encoding. Nothing would work unless someone rewrite it from scratch.
https://github.com/JeffyCN/FFmpeg/issues/4#issuecomment-1179454405
If you just want to use HW accel in jellyfin, my project should be enough https://github.com/jjm2473/ffmpeg-rk/wiki
Note: HW decodeing/encoding works, RGA scale works. No tone mapping supported. Tested on rk3568/rk3588.
Update 2023-11-24: now supports HW mjpeg decoding; rga_scale filter supports setting output pix format, supports yuvj42xp/yuyv422/rgb as input .
@jjm2473 I think mpp kernel driver is a rockchip vendor kernel only thing. Armbian for one has moved to mainline thus as far as I understand I cannot use ffmpeg-rk on an Armbian install.
My understanding is that rockchip has no plan to port mpp drivers upstream. And hantro h264 encoding has been added to the upstream hantro driver. So I believe v4l-request is the way to support upstream kernels.
@prahal No, the MPP drivers not upstreamed. But mainline linux can be patch to supports MPP. Not only MPP drivers, but also RGA drivers shoubld be supported, because some SoC has not enough performance to encoding 4K/30FPS video (for example RK3568, up to 1080p/30FPS encoding), must use RGA to scale video.
I have no experience with v4l. The ffmpeg-rk project is only implemented by referring to RK's old ffmpeg and MPP/RGA development documents, so it depends on related kernel functions.
@jjm2473 https://aur.archlinux.org/packages/rockchip-mpp?all_reqs=1 tells tells MPP only works with rockchip BSP kernel ( latest 5.10 in a develop branch).
Where could I get these MPP and RGA drivers against linux 6.4-rcN?
To me, the issue is not that MPP and RGA are not upstream yet. But that rockchip loos like it has no plan to devote developer time to upstream it. This looks like an internal project for their custom kernel that they develop on a public github to help their customers customize products based on their kernel.
linux 6.4-rcN
I hate to quote a meme but... That's the neat part, you don't!
There has not been a single commit to their kernel repo in forever - let alone a new branch to support anything newer than 5.10.x
In fact, here's my current one on a RK3588 (FriendlyElec NanoPi R6s)
# uname -a
Linux FriendlyWrt 5.10.110 #1 SMP Sat Dec 3 01:25:15 CST 2022 aarch64 GNU/Linux
This corresponds to https://github.com/rockchip-linux/kernel/tree/develop-5.10
linux 6.4-rcN
I hate to quote a meme but... That's the neat part, you don't!
Then should never be part of a linux distribution as not security supported. I wonder why Archlinux ships userspace mpp-git https://aur.archlinux.org/packages/mpp-git.
Sadly v4lrequests development is somewhat stalled especially on the vaapi side (there is a ffmpeg v4lrequests specific patch which is more up to date but not upstreamed either). I do not know about scaling with v4lrequests.
@prahal Check these drivers and patches: https://github.com/istoreos/istoreos/tree/istoreos-22.03/target/linux/rockchip/rk35xx/files/drivers/video/rockchip https://github.com/istoreos/istoreos/tree/istoreos-22.03/target/linux/rockchip/rk35xx/patches-5.10
Maybe someone can port to linux 6.x?
@jjm2473 if it was that easy I hope rockchip would have paid their developers to do the upstreaming. No definitely MPP is for a rockchip fork of the linux kernel. This is not a Linux kernel, it is only based on it. A little like the Windows TCP/IP stack based on BSD code does not make Windows a BSD kernel, except here the fork is only 3 years old. Like you can add hw video acceleration for a BSD stack to jellyfin ffmpeg you can add the Rockchip kernel video acceleration stack support to jellyfin ffmpeg. This will not support for Linux on Rockchip hardware.
I am after an hw acceleration stack for Armbian current and edge, that is Linux per see. So MPP is of no help. Sadly v4l-request upstreaming seems to have stalled. I will try to contribute to it if it is not dead yet. I am especially fond of the v4l-request to vaapi bridge that could enable to share code with the intel vaapi support that is already in (sadly this bridge is the part that looks completely dead development-wise). It looks to me like v4l-request does the decode/encode job (hantro vpu and rockchip hw decoder. But I know little about rockchip RGA code. Might be this part cannot be supported by v4l-request only.
linux 6.4-rcN
I hate to quote a meme but... That's the neat part, you don't!
There has not been a single commit to their kernel repo in forever - let alone a new branch to support anything newer than 5.10.x
In fact, here's my current one on a RK3588 (FriendlyElec NanoPi R6s)
# uname -a Linux FriendlyWrt 5.10.110 #1 SMP Sat Dec 3 01:25:15 CST 2022 aarch64 GNU/LinuxThis corresponds to https://github.com/rockchip-linux/kernel/tree/develop-5.10
Well Friendlyelec just updated their Rockchip branch to 6.1
https://github.com/friendlyarm/kernel-rockchip
This repo is rarely updated.
- https://github.com/rockchip-linux/kernel
Use these instead. This mirror is maintained by Radxa.
- https://gitlab.com/rk3588_linux/rk/kernel/-/commits/develop-6.1
- https://gitlab.com/rk3588_linux/rk/kernel/-/commits/develop-5.10
The current MPP module has two parts: the userspace libmpp.so in the repo and the mpp_service in kernel drivers/video/rockchip/mpp. This framework simplifies the kernel driver and keep most complicated and flexible part in userspace libmpp.so.
We use this framework in consider of our product requirement. We need to support multiple kernel version and have to update custorm's very old version firmware to fix issues. The kernel update is quite hard for already mass-produced firmware so we choose to update libmpp.so in userspace for custerms with different kernel version. This reduce our workload a lot.
About the kernel v4l2 driver it is quit hard to adapt to diffferent user requirements for example fast display on start playing. And also move all video codec protocol handling code to kernel is very hard to be accepted for kernel maintainers.
So base on these historical and realistic consideration we do not develop a common v4l2 like driver in mainline kernel. But for next generation MPP driver we are considering move the whole driver to kernel.
The above quote is from RK developer. They will not spend time working on the upstream v4l2-request. Because this interface cannot meet their actual needs.
To be honest, I don't have a good impression of v4l2-request, given that the RK3399 released 7 years ago has not received V4L2 encoding support yet. And actually the encoder API is not finalized. Not to mention the RGA part.
Then I got in touch with RK developer to see what they thought about native Libva/VA-API support. He said they would look into it.