Joshua-Riek
Orange pi 5 plus HDMI IN native V4L2 support really needed (OBS crashes)
Thank you for your tremendous efforts!!! I'm trying to build a re-streaming platform out of orange pi 5 plus with 24.04 Ubuntu from you. But there is a bummber issue like OBS and other apps cannot open HDMI in via v4l2 propertly due to driver IOCTL missing.
So i found another fork where it seems to me all fixed and hdmirx driver patched for rockchip and v4l2 compliance runs with no errors. https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/issues/4#note_152133 The core issue which crashes OBS is lack of VIDIOC_G_PARM ioctl.
Could you please merge those chages with you build?
Grabbing HDMI IN with gstreamer or ffmpeg is working but i'm getting 30 fps max at 4K resolution and cannot change it no matter what setting i'm trying to apply. I need to get 2K in 60 FPS. I think this orange pi is capable of doing it with no effors (spec says 4K 60fps but nobody can get it from HDMI IN so far). It would be great to just plug and play HDMI IN via V4L2 directly to OBS and check what fps will i get then.
Thank you!
I'm trying to build the media module right now and test in on my Orange PI 5 PLUS. But it might take a while as soon as i'm really new in kernel drivers building. Struggling with finding proper way to override kernel media module with my module at working ubuntu 24.04.
What did i do:
- built a m2m-deinterlace.ko (i'm pretty sure this is not what needed). Added VIDIOC_G_PARM handler to drivers/media/platform/rockchip/hdmirx/rk_hdmirx.c.
- installed it to /lib/modules/6.1.0-1013-rockchip/extra
- reboot
- nothing changed.
I really need a good push in right direction =)
If you have the kernel build dependencies installed correctly cd into your kernel tree and enter the below to re-build the kernel. You can then install the deb package.
# Setup env
export $(dpkg-architecture -aarm64)
export CROSS_COMPILE=aarch64-linux-gnu-
export CC=aarch64-linux-gnu-gcc
export LANG=C
# Compile the kernel into a deb package
fakeroot debian/rules clean binary-headers binary-rockchip do_mainline_build=true
I can try to test on my end in a few hours, i'm currently working on some U-Boot code for some new Radxa boards.
I am quite sure that in order to get hdmi in to work in obs studio we would need panthor. This is what amazingfate claim. So, should be pinned here.
Guys. Sorry for delay. I had to do a slight detour to GPIO and FAN settings, because during kernel compilation my orange pi was trying to melt ... So, the progress exist. I'v managed to reduce amount of v4l2-compliance errors from 9 to 7 as of today... OBS still crashes though...
I am quite sure that in order to get hdmi in to work in obs studio we would need panthor. This is what amazingfate claim. So, should be pinned here.
Well according to the article new GPU driver will be merged with kernel 6.10 (we expect this driver to land in Linux v6.10). But we are sitting on 6.1. I think it will take years to get there... Do you think enabling HDMI IN v4l2 full support completerly impossible with current panfrost driver? Even if i'll fix hdmirx driver for rk3855?
just let it here for future reference https://www.collabora.com/news-and-blog/news-and-events/release-the-panthor.html
I mean, currently, hdmi in performs really bad in obs wheb used with v4l2 (at least when it worked). The only real way to use it is with a gst pipeline, and that's inside a windowd, not in obs itself. Panthor is meant for mainline, but was backpoorted by boogie to bsp 6.1
Ok this is my bes effort after 2 weeks of sufferring without sleep, food and life itself ... v4l2-compliance shows 0 errors and 0 warnings! But driver still throttling and dropping frames -((( so i'm getting same 4k30 or 1080p30 no more... OBS still crashes also ... I really need someboody to look at my attemt here https://github.com/MaximKurakin/linux-rockchip.git
I was making test like this: gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! fpsdisplaysink showing only unstable 30 fps at 4K and even at 1080p60 this test also shows 30 FPS! why? external source gives solid 60 fps.
ffmpeg -f v4l2 -video_size 3920x2160 -i /dev/video0 -pix_fmt yuv420p -vf "format=yuv420p" -framerate 60 -c:v rawvideo -vsync 1 -f sdl "SDL output" shows broken picture repeting screen 3 time in a row with broken colors (Trying to reach 0 errors in v4l2-compliance I messed up with yuv420p colospace i think)
I have tried to merge synopsis driver with rockchip because the source code is almost the same in general... But i'm week in DMA flags, bits and fencing... I need somebody more talanted and experienced to look at it. Any help is very much appreciated! I think if we can solve DMA sync issues we will finally fix rockchip driver for good and will get 4k60 as entitled in orange pi 5 plus specification... Nobody touched this driver since 2021... Let's fix it up all together!
I would love to help, but I am not capable. Did you try to contact amazingfate? The only way I got decent hdmi in was with ximagesynk streaming pipelines and obs recording my desktop, at even then, I never got 1080p60 out of it. He claimed tha tin order to really use it inside obs, so, as a v4l2 source, we had to use panthor and even then it would need more work. You should at least co tact him and boogie at radxa discord, or armbian one.
@MaximKurakin Did you happen to test HDMI audio input with the Collabora code? It was my understanding that they didn't support it yet.
Also, you might want to look at the obs-gstreamer plugin. If performance is bad, you might be doing colour format conversion on the CPU rather than RGA. I've never tried it directly in OBS, just in gstreamer directly.
Collabora driver doesn't support audio and HDCP. Thats why I went into madness of merging current rockchip driver with it. I wanted to take only good working parts from both of them and create new complianced driver with full support of video, audio, CEC and HDCP support. I've tried everything I could google =) gstreamer fpsdisplaysink is my main testing tool. The problem is not in the external tools like ffplay, vlc, gstreamer ... the driver itself messes up with DMA timings and my 60 fps turns into 30 because drivers is shouting about massive frame drops internally.
In debug mode drivers gives something like:
[ 1632.249161] fdee0000.hdmirx-controller: dma_irq st1:0x80, st13:2160
[ 1632.249164] fdee0000.hdmirx-controller: dma_idle_int_handler: next_buf NULL, skip vb_done!
[ 1632.257834] fdee0000.hdmirx-controller: dma_irq st1:0x100, st13:1081
[ 1632.257844] fdee0000.hdmirx-controller: line_flag_int_handler: curr_fd:0
[ 1632.257847] fdee0000.hdmirx-controller: line_flag_int_handler: next_fd:24887296
[ 1632.265836] fdee0000.hdmirx-controller: dma_irq st1:0x80, st13:2160
[ 1632.265847] fdee0000.hdmirx-controller: vb_done fd:0
[ 1632.273395] fdee0000.hdmirx-controller: qbuf fd:37330944
[ 1632.274504] fdee0000.hdmirx-controller: dma_irq st1:0x100, st13:1081
[ 1632.274510] fdee0000.hdmirx-controller: line_flag_int_handler: curr_fd:24887296
[ 1632.274512] fdee0000.hdmirx-controller: line_flag_int_handler: next_fd:37330944
[ 1632.282495] fdee0000.hdmirx-controller: dma_irq st1:0x80, st13:2160
[ 1632.282504] fdee0000.hdmirx-controller: vb_done fd:24887296
[ 1632.291167] fdee0000.hdmirx-controller: dma_irq st1:0x100, st13:1081
[ 1632.291174] fdee0000.hdmirx-controller: line_flag_int_handler: next_buf NULL, drop the frame!
[ 1632.291176] fdee0000.hdmirx-controller: line_flag_int_handler: curr_fd:37330944
[ 1632.299162] fdee0000.hdmirx-controller: dma_irq st1:0x80, st13:2160
[ 1632.299165] fdee0000.hdmirx-controller: dma_idle_int_handler: next_buf NULL, skip vb_done!
[ 1632.303915] fdee0000.hdmirx-controller: qbuf fd:12443648
[ 1632.307829] fdee0000.hdmirx-controller: dma_irq st1:0x100, st13:1080
[ 1632.307832] fdee0000.hdmirx-controller: line_flag_int_handler: curr_fd:37330944
[ 1632.307834] fdee0000.hdmirx-controller: line_flag_int_handler: next_fd:12443648
[ 1632.315828] fdee0000.hdmirx-controller: dma_irq st1:0x80, st13:2160
[ 1632.315833] fdee0000.hdmirx-controller: vb_done fd:37330944
@MaximKurakin Did you happen to test HDMI audio input with the Collabora code? It was my understanding that they didn't support it yet.
Also, you might want to look at the obs-gstreamer plugin. If performance is bad, you might be doing colour format conversion on the CPU rather than RGA. I've never tried it directly in OBS, just in gstreamer directly.
I tried this plugin today (had a hard time with unclear dependencies during buld). yes i can capture gstreamer directly to obs now. but fps is still 30 and tonns of dropped frames -( Thank you for you advice anyway!
Also I tried obs beam plugin. (had very sophisticated intercource with build process for arm64). Yes it works but device cannot keep up with 2k 60fps at any jpeg compression quality even in quality 1!!!! black screen always flashing from time to time. All equipment has 2.5 Gbps caps but this orange pi 5 plus really cant keep up with the massive frames comming from network... I don't know if anybody even tried to launch obs-beam on orange pi ... but I did and it doesn't work as expected -((( platform simply doen't have enough performance
You should at least co tact him and boogie at radxa discord, or armbian one.
Contacted. He only introduced : media: rockchip: hdmirx: add VIDIOC_G_PARM hack fix for Armbian. BUt this fix doesn' work for Jushua Riek build. I tested it already. https://github.com/armbian/linux-rockchip/commit/390072479592339b732a0aa52bdaa14381f19116
But he gave me hope with armbian embedded panthor driver support out of the box. SO, i'm going to give it a spin later today. Maybe with panthor driver overal graphical performance will be better? https://github.com/armbian/linux-rockchip/pull/171
Panthor does not have better performance and has quite a few bugs and problems when playing games.
To me, it has better performance,.overall. but even so, it has better features, and will have much better ones down the road. This is not about performance, but feautres
Ok. No more ARMBIAN for me!!!! It freezed 3 times in 30 minuts. HDMIRX driver exactly the sam as in Josua build ... 30 fps max. 9 compliance errors an 6 warnings. Morover! OBS doesn't recognize new panthor graphics and doesn't start . THis is a dead end to me. I want to say great thank you to Joshua for this stable and glitch free build. I'll stick with it for a while and will try to contact rk chip developers directly. Because fore the fu8ck sake how the f888ck do you pass unit test for hdmirx driver if it throttles and v4l2-compliance shows 9 errors ? Damn...
Obs doesnt start bc it needs PAN_MESA_DEBUG=gl3, it's probably forced on panfork by default
I understand your suffering! Hahah yeah, hdmi in is only really usable from android as today.
I don't give up with it. I'm just taking a pause to collect more information about HDMI protocol and rk388 chip memory access features. But after 2.5 weeks reading tons of web-sites and manuals ... I need vacation
I don't give up with it. I'm just taking a pause to collect more information about HDMI protocol and rk388 chip memory access features. But after 2.5 weeks reading tons of web-sites and manuals ... I need vacation
Please contact me at armbian discord. Microlinux(salva)
I'm definitely interested in collaborating on this, as I've been working with Rockchip SoCs for about a year and a half now for mobile livestreaming.
I've generally found the HDMI input to work well enough, except when it doesn't. And it can be frustrating when it doesn't.
Couple thoughts:
Early on, I was seeing pretty choppy performance, dropped frames, etc, but it seemed like at least some of that wasn't the HDMI input itself's fault, but a downstream element not being fast enough. In my case, the HDMI input was using a pixel format the encoder I was using didn't support, so I need to use a videoconvert element in between the capture and encoder.
By default, this seems to be running on the CPU, rather than the RGA hardware (which does support some pixel format conversions). When running on the CPU, it wasn't fast enough to always keep up, especially with 2160p sources, so it'd drop frames as buffers filled up. I was able to get good performance by setting the GST_VIDEO_CONVERT_USE_RGA environment variable to 1, which uses the RGA for hardware acceleration.
The drivers seem very fragile in their current state. Slight disturbances to the kernel, etc, seem to break things. I've had audio just stop working multiple times, and at least at one point, the colours went all weird.
When a briefly looked at OBS support, I was seeing issues due to multi-planar formats. I haven't looked closely, but this patchset may help solve some of those (or other) issues with the RGA and multi-planar formats: https://patchwork.kernel.org/project/linux-media/cover/20230914-rockchip-rga-multiplanar-v1-0-abfd77260ae3@pengutronix.de/
Nice, I used that env var on my pipelines too and yes, obs seems to not support multiplannar formats. https://github.com/neofeo/BOX86-BOX64-WINEx86-TUTORIAL/blob/main/gst/hdmi_in.sh
Hmm . This GST_VIDEO_CONVERT_USE_RGA didn't help me. Same frame drops and 30 fps max -((((
More insteresting is OBS crach logs:
sendmsg(11, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\4\0\0\0\0\0\f\08\0\0\0\4\0\0\0\0\0\f\0;\0\0\0\4\0\0\0\1\0\f\0"..., iov_len=708}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 708
futex(0xaaaae62622fc, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xaaaae62622a0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xaaaae6526c58, FUTEX_WAKE_PRIVATE, 1) = 1
ppoll([{fd=12, events=POLLIN}, {fd=14, events=POLLIN}, {fd=21, events=POLLIN}, {fd=29, events=POLLIN}, {fd=33, events=POLLIN}], 5, {tv_sec=0, tv_nsec=0}, NULL, 0) = 1 ([{fd=14, revents=POLLIN}], left {tv_sec=0, tv_nsec=0})
read(14, "1\0\0\0\0\0\0\0", 8) = 8
write(14, "\1\0\0\0\0\0\0\0", 8) = 8
**openat(AT_FDCWD, "/dev/v4l/by-id/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)**
writev(2, [{iov_base="free(): invalid pointer", iov_len=23}, {iov_base="\n", iov_len=1}], 2free(): invalid pointer
) = 24
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffb60b1000
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
gettid() = 5117
getpid() = 5117
tgkill(5117, 5117, SIGABRT) = 0
--- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=5117, si_uid=1000} ---
+++ killed by SIGABRT (core dumped) +++
Aborted (core dumped)
and same time hdmirx shows only this one line:
[ 826.694132] fdee0000.hdmirx-controller: hb
[ 827.704136] fdee0000.hdmirx-controller: hb
**[ 827.897675] fdee0000.hdmirx-controller: tx_5v_power_present: 1**
[ 828.720816] fdee0000.hdmirx-controller: hb
[ 829.734162] fdee0000.hdmirx-controller: hb
I hope it will give somebody a hint where is the issue
I haven't looked closely, but this patchset may help solve some of those (or other) issues with the RGA and multi-planar formats: https://patchwork.kernel.org/project/linux-media/cover/20230914-rockchip-rga-multiplanar-v1-0-abfd77260ae3@pengutronix.de/
I'm on it. Will test it in a couple hours...
I haven't looked closely, but this patchset may help solve some of those (or other) issues with the RGA and multi-planar formats: https://patchwork.kernel.org/project/linux-media/cover/20230914-rockchip-rga-multiplanar-v1-0-abfd77260ae3@pengutronix.de/
I'm on it. Will test it in a couple hours...
Nope... nothing changed. The most surprizing part is that i'm grabbing 1080p60 and getting 30 fps. Then I set 30 hz to the HDMI 2 display out (i'm using orange pi 5 plus as a source and as an input just made a cable loop hdmi2 to hdmi in) and grab it - i'm getting 20 fps... oddd
More research comming up:
I stopped totruring hdmi in and decided to make a clean perf test of my SBC caps. Here is the result:
- test with 4K30 gave me more or less stable 30 fps with low drops.
GST_VIDEO_CONVERT_USE_RGA=1 gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,framerate=30/1 ! fpsdisplaysink - test with 4K60 killed my board and gave me 0 fps ! all frames were dropped! I wonder why?
GST_VIDEO_CONVERT_USE_RGA=1 gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,framerate=60/1 ! fpsdisplaysink - test with 1080p120 was surprisingly floppy. FPS were jumping betwen 110 and 60 fps. with massive drops. WTF?
GST_VIDEO_CONVERT_USE_RGA=1 gst-launch-1.0 videotestsrc ! video/x-raw,width=1920,height=1080,framerate=120/1 ! fpsdisplaysink
I did the test wrong way? Or it's my orange pi 5 plus actual limits ?
@MaximKurakin You might have better luck running videotestsrc is-live=true. I know it sometimes matters, but might not here.
I have previously been able for capture from a 4k60 camera and encode and stream, but it seems highly dependent on kernel (+patches), MPP, RGA, etc libraries.
Nope. Same behaviour. And I'm testing at clean build 1016 from you. No patches nothing. =((( Myabe orange PI 5 is a piece of sh**t? and I have better to look at radxa boards?
I don't think it's a hardware issue, because I've definitely had things working well, but then I'll update and things will break again. I'm presently in a state where the 6.1 kernel has no audio but captures+encodes video great, and the 5.10 kernel segfaults whenever I use the RGA to do format conversions, but does audio fine.
Though the HDMI input on the OPi5+ does have issues with HDMI backpowering, so that could point to other implementation issues existing. Who knows...