[BUG] rpi4 video crashing with certain addons

[ghost]

Describe the bug

when playing a twitch stream via the addon the video driver seems to crash. this causes the stream to freeze and needs either a process restart or hard reboot. system does not accept inputs, but ssh still available.

To Reproduce

Steps to reproduce the behavior:

1. install twitch add on
2. start any stream
3. observe freeze when any ad (or some scene transitions) happen
4. kill process or restart system
5. journalctl -xe to see errors listed below which repeat until process/system restart

Informations

• LE Version: 10.0.2
• Hardware Platform: rpi4

Log file

Jul 11 19:17:51 LibreELEC kernel: bcm2835_mmal_vchiq: dropping event 0x48434645
Jul 11 19:17:57 LibreELEC kernel: ------------[ cut here ]------------
Jul 11 19:17:57 LibreELEC kernel: WARNING: CPU: 0 PID: 4187 at __vb2_queue_cancel+0x1d4/0x240 [videobuf2_common]
Jul 11 19:17:57 LibreELEC kernel: Modules linked in: hci_uart btbcm bluetooth ecdh_generic ecc 8021q brcmfmac brcmutil cfg80211 bcm2835_codec(C) rpivid_hevc(C) bcm2835_isp(C) bcm2835_mmal_vchiq(C) v4l2_mem2mem videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common rfkill videodev bcm2835_gpiomem mc n
Jul 11 19:17:57 LibreELEC kernel: CPU: 0 PID: 4187 Comm: VideoPlayer Tainted: G        WC        5.10.95 #1
Jul 11 19:17:57 LibreELEC kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
Jul 11 19:17:57 LibreELEC kernel: pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--)
Jul 11 19:17:57 LibreELEC kernel: pc : __vb2_queue_cancel+0x1d4/0x240 [videobuf2_common]
Jul 11 19:17:57 LibreELEC kernel: lr : __vb2_queue_cancel+0x40/0x240 [videobuf2_common]
Jul 11 19:17:57 LibreELEC kernel: sp : ffffffc0136f3ab0
Jul 11 19:17:57 LibreELEC kernel: x29: ffffffc0136f3ab0 x28: ffffff8054d86000
Jul 11 19:17:57 LibreELEC kernel: x27: 0000000040045613 x26: ffffffc008d92aa8
Jul 11 19:17:57 LibreELEC kernel: x25: 0000000000000000 x24: ffffffc0136f3cd8
Jul 11 19:17:57 LibreELEC kernel: x23: ffffff80447dce30 x22: ffffff8043ab0828
Jul 11 19:17:57 LibreELEC kernel: x21: 0000000000000009 x20: ffffff8043ab0828
Jul 11 19:17:57 LibreELEC kernel: x19: ffffff8043ab0828 x18: 0000000000000000
Jul 11 19:17:57 LibreELEC kernel: x17: 0000000000000000 x16: 0000000000000000
Jul 11 19:17:57 LibreELEC kernel: x15: 0000000000000004 x14: ffffff8048fc0030
Jul 11 19:17:57 LibreELEC kernel: x13: 0000000000000000 x12: ffffff804ea46cc8
Jul 11 19:17:57 LibreELEC kernel: x11: ffffff804ea46b58 x10: ffffff804ea46b5a
Jul 11 19:17:57 LibreELEC kernel: x9 : ffffffc008d78844 x8 : ffffff804ea46b80
Jul 11 19:17:57 LibreELEC kernel: x7 : 0000000000000001 x6 : 0000000000000000
Jul 11 19:17:57 LibreELEC kernel: x5 : ffffff80447dd01c x4 : 0000000000000000
Jul 11 19:17:57 LibreELEC kernel: x3 : ffffff8048fc0008 x2 : 0000000000000000
Jul 11 19:17:57 LibreELEC kernel: x1 : 00000000000055ac x0 : 0000000000000001
Jul 11 19:17:57 LibreELEC kernel: Call trace:
Jul 11 19:17:57 LibreELEC kernel:  __vb2_queue_cancel+0x1d4/0x240 [videobuf2_common]
Jul 11 19:17:57 LibreELEC kernel:  vb2_core_streamoff+0x30/0xd0 [videobuf2_common]
Jul 11 19:17:57 LibreELEC kernel:  vb2_streamoff+0x28/0x80 [videobuf2_v4l2]
Jul 11 19:17:57 LibreELEC kernel:  v4l2_m2m_streamoff+0x48/0x150 [v4l2_mem2mem]
Jul 11 19:17:57 LibreELEC kernel:  v4l2_m2m_ioctl_streamoff+0x24/0x30 [v4l2_mem2mem]
Jul 11 19:17:57 LibreELEC kernel:  v4l_streamoff+0x30/0x40 [videodev]
Jul 11 19:17:57 LibreELEC kernel:  __video_do_ioctl+0x190/0x3f0 [videodev]
Jul 11 19:17:57 LibreELEC kernel:  video_usercopy+0x1b0/0x530 [videodev]
Jul 11 19:17:57 LibreELEC kernel:  video_ioctl2+0x24/0x50 [videodev]
Jul 11 19:17:57 LibreELEC kernel:  v4l2_ioctl+0x4c/0x70 [videodev]
Jul 11 19:17:57 LibreELEC kernel:  v4l2_compat_ioctl32+0x1c4/0x1ac0 [videodev]
Jul 11 19:17:57 LibreELEC kernel:  __arm64_compat_sys_ioctl+0xb4/0x1bc
Jul 11 19:17:57 LibreELEC kernel:  el0_svc_common.constprop.0+0x84/0x1e0
Jul 11 19:17:57 LibreELEC kernel:  do_el0_svc_compat+0x28/0x60
Jul 11 19:17:57 LibreELEC kernel:  el0_svc_compat+0x20/0x30
Jul 11 19:17:57 LibreELEC kernel:  el0_sync_compat_handler+0x90/0x160
Jul 11 19:17:57 LibreELEC kernel:  el0_sync_compat+0x184/0x1c0
Jul 11 19:17:57 LibreELEC kernel: ---[ end trace 32c40947968c5e80 ]---

Additional context

https://github.com/anxdpanic/plugin.video.twitch/issues/648 - related issue reported to the addon repo

[ghost]

i tested a generic install of libreelec in virtualbox and it seemed to work. likely something rpi4 driver specific.

[HiassofT]

Please test with the latest LE10 nightly build from there https://test.libreelec.tv/10.0/RPi/

If the issue persists post a full debug log

[ghost]

@HiassofT i dont see anything in that directory. should i try a 11 nightly? what command should i run for the proper debug log for kodi? that is the whole section from journalctl, nothing before or after really

[HiassofT]

@portno12 ah, sorry, bad timing. We just switched to a new server and new builds should (hopefully) start appearing soon.

Better wait another one or two days and then test with a LE10 nightly.

If you have a spare SD card a test with LE11 would also be interesting

[HiassofT]

LE10 RPi nightly builds just finished and are up on https://test.libreelec.tv/10.0/RPi/

[ghost]

@HiassofT i get the same error on the 10 nightly. 11 nightly also has an issue but i cannot even ssh to the system to check any logs. can you tell me which debug logs you need and how to generate them? the error i posted above just came from journalctl since its a kernel error, not kodi application error.

[HiassofT]

Enable debug logging in kodi, reboot the RPi, reproduce the problem and then ssh in and run "pastekodi"

We need the kodi log as well as it'll contain some important information about the video/stream. Unless it's an obvious issue we'll need a file to reproduce, analyse and fix the issue

[ghost]

i managed to get the debug log but it doesn't seem to want to paste right with pastekodi so i tried the -c option and other pastebins but it is still too large. is there an email i can send it to or another way to provide the log? i was able to get it immediately after a crash.

[HiassofT]

Just send it to me by email: [email protected] (message limit is about 20MB IIRC)

[ghost]

email should be on its way. thank you. i did also do a test and the same happens with raspbian so i am not sure its libreelec specific.

[HiassofT]

Thanks, got the log!

But it shows you are running LE 10.0.2, not the latest LE10 nightly. Please install the nightly and post (or email me) another full log running that.

If you have a spare SD card please install the latest LE11 nightly on it and test with that as well (and post a log from that, too, if the issue persists)

[ghost]

ah sorry. i forgot i swapped back to stable. give me a couple days and i will email you the new logs from both 10 and 11 nightlies.

[ghost]

i was able to grab the logs tonight and sent them along for both 10 and 11 nightly.

[HiassofT]

Thanks a lot for testing and the logs!

Good to know the latest versions have the same issue, it indeed looks like an RPi video decoder bug. I've informed RPi devs about it

[ghost]

thanks. in the meantime is there a way i can keep nightly updated or should i drop back to stable?

[HiassofT]

I'd recommend staying on LE10 nightly as it contains several other important fixes that were added after LE10.0.2.

You can easily update to newer LE10 nightly builds or the next official release by copying the img.gz to the update share/folder and rebooting.

[ghost]

i did a quick test with a plain arch install and kodi and got the same error. except it didnt log anything and crashed the whole system to the point trying to ssh wasnt even possible. the same symptom on latest le11 nightly, when crash happens a hard reboot is required

[andrewgreenwood]

I believe I may have the same problem, but on a Raspberry Pi 3 and with the TVHeadend PVR add-on.

Previously, I'd been running 9.2 which had been pretty solid. I updated to 10.0.2 a couple of weeks ago and started experiencing freezes, then tried the latest nightly shortly after and the problem remains. Usually I can SSH in and force a restart, but on one occasion I could only ping the device and had to power cycle it.

I've got debug logging turned on and have been occasionally checking the Kodi logs and "dmesg" output (had not thought of using "journalctl -xe" but I'll do this next time.) I've saved a copy of the Kodi log from the previous freeze where I had to power cycle and another from where it had frozen and I managed to restart it - are you happy for me to e-mail these to you @HiassofT ?

When it freezes up and I'm able to SSH in, dmesg only reports this over and over:

[56283.208948] [drm] Resetting GPU.
[56284.195627] [drm] Resetting GPU.
[56285.208959] [drm] Resetting GPU.
[56286.195658] [drm] Resetting GPU.

Sometimes, whilst Kodi seems to be working fine, the dmesg output occasionally contains this (I'm unsure if maybe this is a symptom leading up to the eventual hang, since both seem video-related):

[158067.505385] ------------[ cut here ]------------
[158067.505405] WARNING: CPU: 3 PID: 5544 at drivers/media/common/videobuf2/videobuf2-core.c:2000 __vb2_queue_cancel+0x244/0x2bc [videobuf2_common]
[158067.505439] Modules linked in: hci_uart btbcm bluetooth ecdh_generic ecc 8021q brcmfmac brcmutil bcm2835_codec(C) cfg80211 bcm2835_isp(C) v4l2_mem2mem videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common aes_arm_bs rfkill videodev crypto_simd cryptd bcm2835_gpiomem snd_bcm2835(C) mc pkcs8_key_parser fuse
[158067.505609] CPU: 3 PID: 5544 Comm: VideoPlayer Tainted: G        WC         6.0.0 #1
[158067.505618] Hardware name: BCM2835
[158067.505624] Backtrace:
[158067.505631]  dump_backtrace from show_stack+0x20/0x24
[158067.505653]  r7:7f0aae64 r6:00000000 r5:80d75404 r4:68070013
[158067.505658]  show_stack from dump_stack_lvl+0x98/0xb4
[158067.505671]  dump_stack_lvl from dump_stack+0x18/0x1c
[158067.505687]  r7:7f0aae64 r6:00000009 r5:000007d0 r4:7f0add50
[158067.505691]  dump_stack from __warn+0xd4/0x138
[158067.505706]  __warn from warn_slowpath_fmt+0x70/0xe0
[158067.505721]  r8:7f0aae64 r7:00000009 r6:000007d0 r5:7f0add50 r4:00000000
[158067.505725]  warn_slowpath_fmt from __vb2_queue_cancel+0x244/0x2bc [videobuf2_common]
[158067.505748]  r8:935f3014 r7:8456d3d0 r6:935f3014 r5:00000009 r4:935f3014
[158067.505753]  __vb2_queue_cancel [videobuf2_common] from vb2_core_streamoff+0x28/0xb4 [videobuf2_common]
[158067.505782]  r10:00000000 r9:7f0729d4 r8:7f0581f0 r7:8456d3d0 r6:935f3014 r5:00000009
[158067.505788]  r4:935f3014 r3:00000009
[158067.505792]  vb2_core_streamoff [videobuf2_common] from vb2_streamoff+0x28/0x64 [videobuf2_v4l2]
[158067.505818]  r5:00000009 r4:935f3000
[158067.505823]  vb2_streamoff [videobuf2_v4l2] from v4l2_m2m_streamoff+0x50/0x11c [v4l2_mem2mem]
[158067.505843]  v4l2_m2m_streamoff [v4l2_mem2mem] from v4l2_m2m_ioctl_streamoff+0x20/0x24 [v4l2_mem2mem]
[158067.505866]  r10:00000000 r9:7f0729d4 r8:7f0581f0 r7:8456d3d0 r6:00000000 r5:40045613
[158067.505871]  r4:7f0d5d00
[158067.505875]  v4l2_m2m_ioctl_streamoff [v4l2_mem2mem] from v4l_streamoff+0x2c/0x30 [videodev]
[158067.505926]  v4l_streamoff [videodev] from __video_do_ioctl+0x234/0x458 [videodev]
[158067.505988]  r5:40045613 r4:8456d0a0
[158067.505993]  __video_do_ioctl [videodev] from video_usercopy+0x2c8/0x5f8 [videodev]
[158067.506058]  r10:00000000 r9:bc299e3c r8:40045613 r7:00000000 r6:6dc595a0 r5:40045613
[158067.506063]  r4:00000000
[158067.506069]  video_usercopy [videodev] from video_ioctl2+0x20/0x24 [videodev]
[158067.506133]  r10:82dfc800 r9:00000035 r8:89c99300 r7:4c6cf884 r6:6dc595a0 r5:89c99301
[158067.506139]  r4:7f05e6d0
[158067.506143]  video_ioctl2 [videodev] from v4l2_ioctl+0x4c/0x60 [videodev]
[158067.506204]  v4l2_ioctl [videodev] from sys_ioctl+0x3f0/0x9f4
[158067.506242]  r5:89c99301 r4:40045613
[158067.506246]  sys_ioctl from ret_fast_syscall+0x0/0x1c
[158067.506258] Exception stack(0xbc299fa8 to 0xbc299ff0)
[158067.506270] 9fa0:                   6dc08fa8 40045613 00000035 40045613 4c6cf884 4c6cf868
[158067.506279] 9fc0: 6dc08fa8 40045613 6dc595a0 00000036 6dc595a0 10624dd3 00000000 4c6cf980
[158067.506287] 9fe0: 760f920c 4c6cf858 75d49b90 747edf6c
[158067.506296]  r10:00000036 r9:82dfc800 r8:80100268 r7:00000036 r6:6dc595a0 r5:40045613
[158067.506304]  r4:6dc08fa8
[158067.506309] ---[ end trace 0000000000000000 ]---

The stack trace here looks similar to the one in the OP.

[heitbaum]

Please confirm if this is still an issue in LE11.0.3 and/or LE12 nightlies.

[thomasb22]

I have same problem with Twitch on Raspberry Pi 4 and LE 11.0.3.

Log without inputstream.adaptive: kodi.old.log

Log with inputstream.adaptive: kodi.log

[thomasb22]

Another test, always with inputstream.adaptive:

Journalctl: 2-journalctl-libreelec-crash-twitch-ads.log

Kodi: 2-kodi-libreelec-crash-twitch-ads.log

Logs cut at moment of crash (because network crash too and I copy/past from SSH connection).

[heitbaum]

@thomasb22 could you please retest with LE12 nightlies. These are using the updated 6.6 LTS kernel and where the development teams from RPi, LE and Kodi are working on.

[thomasb22]

@heitbaum Issue is always here with the last nightly (20231204).

[heitbaum]

Added le12 label

[Haui1112]

Just stumbled upon this using 11.0.6 on an Rpi4. Same issue. Is this related to drm? Seems similar to an issue peeps have reported with arm notebooks with netflix pretty recently iirc.

Are there any workarounds? Its also not in every stream for me, just certain ones. Maybe with music or a lot going on.

[heitbaum]

Just stumbled upon this using 11.0.6 on an Rpi4. Same issue. Is this related to drm? Seems similar to an issue peeps have reported with arm notebooks with netflix pretty recently iirc.

Are there any workarounds? Its also not in every stream for me, just certain ones. Maybe with music or a lot going on.

@Haui1112 for support please go to https://forum.libreelec.tv/ and share your logs.

[heitbaum]

just checking in to see if this is still an issue - please test 20240311-86dba4b images

[heitbaum]

Closing as inactive. Happy to open again - but will need current debug logs from LE12 20240311-86dba4b or newer, ideally a reproducible case. Using the forum for support may get more answers as others may have seen the same issue.