Serial port not working properly after kernel upgrade

[mariusmotea]

Describe the bug

I have a PI5 used for my home stuff (HomeAssistant, Zigbee2MQTT, DiyHue, etc), all running in docker containers. Two days ago i notice more than 100 package updates so i decided to update all and reboot. After few minutes i discovered the lights where not working anymore so i begin to troubleshoot. I notice in zigbee2mqtt configuration the serial interface was pointing to /dev/ttyAMA1 and this was not present anymore on the OS so i replace it with the new name /dev/ttyAMA0. After this the zigbee2mqtt was not crashing anymore but it hang right after it start to connect to the zigbee controller. I performed the following test in order to isolate the problem:

• replacing the zigbee controller (i have 4 in total) => no change
• downgrade docker, containerd => no change
• flashing the zigbee controller firmware using the same serial interface was successful => the serial port was correctly identified and at last is partially working
• test the serial port using the loop method at the 115200 baud rate, the same used by zigbee2mqtt => test was successfully
• connect one zigbee controller to an USB to serial adapter => zigbee2mqtt started and it work as expected.

I spent about 8 hours trying to find a solution to this problem and i know it sound wired but i cannot find an explanation why connecting the zigbee controller to an USB adapter (3.3v TTL) is working fine while when i connect it directly to the pi serial interface i can only flash the firmware but i cannot use it and this is happening exactly after i update the packages on the PI.

Steps to reproduce the behaviour

Connect a TI 2652P zigbee conectoller to Raspberry PI 5 serial interface and start Zigbe2MQTT

Device (s)

Raspberry Pi 5

System

cat /etc/rpi-issue Raspberry Pi reference 2023-12-05 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 70cd6f2a1e34d07f5cba7047aea5b92457372e05, stage4

sudo vcgencmd version 2024/07/30 15:25:46 Copyright (c) 2012 Broadcom version 790da7ef (release) (embedded)

uname -a Linux raspberrypi5 6.6.47+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.47-1+rpt1 (2024-09-02) aarch64 GNU/Linux

Logs

No response

Additional context

No response

[mariusmotea]

Today i made a test with uart3 and uart4, same result like with uart0, the application freeze. I also installed zigbee2mqtt on the host to exclude a problem with docker.

[mariusmotea]

I think i can confirm now that this is a kernel bug. I installed Raspberry PI OS on a new SD card using Imager tool and serial connection to zigbee controller was working fine. The interface rename was not the root issue since the new name (ttyAMA0) was present.

pi@raspberrypi:~ $ uname -r
6.6.31+rpt-rpi-2712

I made it to not work only by updating the kernel.

pi@raspberrypi:~ $ sudo apt install linux-image-rpi-v8
------------------------------
The following NEW packages will be installed:
  linux-headers-6.6.47+rpt-common-rpi linux-headers-6.6.47+rpt-rpi-2712 linux-headers-6.6.47+rpt-rpi-v8 linux-image-6.6.47+rpt-rpi-2712 linux-image-6.6.47+rpt-rpi-v8 linux-kbuild-6.6.47+rpt
The following packages will be upgraded:
  linux-headers-rpi-2712 linux-headers-rpi-v8 linux-image-rpi-2712 linux-image-rpi-v8 linux-libc-dev
--------------------------------------------
Setting up linux-kbuild-6.6.47+rpt (1:6.6.47-1+rpt1) ...
Setting up linux-headers-6.6.47+rpt-common-rpi (1:6.6.47-1+rpt1) ...
Setting up linux-libc-dev (1:6.6.47-1+rpt1) ...
Setting up linux-headers-6.6.47+rpt-rpi-v8 (1:6.6.47-1+rpt1) ...
Setting up linux-image-rpi-2712 (1:6.6.47-1+rpt1) ...
Setting up linux-headers-6.6.47+rpt-rpi-2712 (1:6.6.47-1+rpt1) ...
Setting up linux-headers-rpi-2712 (1:6.6.47-1+rpt1) ...
Setting up linux-image-rpi-v8 (1:6.6.47-1+rpt1) ...
Setting up linux-headers-rpi-v8 (1:6.6.47-1+rpt1) ...
---------------------------
pi@raspberrypi:~ $ sudo reboot

[pelwell]

Thanks - that's a useful bit of differential diagnosis which does implicate a recent kernel change. The problem is going to be assembling a system that can reproduce the problem without requiring significant investments of time or money.

Are there any bits of your system that aren't required? For example, is it necessary for you to containerise Zigbee2MQTT? In other words, what is the minimum (hardware and software) we would need to see the fault?

[mariusmotea]

Hi. Not sure if the issue is happening with all ZigBee2MQTT controllers from Texas Instruments, but mine is TI CC2652P flashed with Z-Stack firmware using this method (without --bootloader-sonoff-usb argument). There are some adapters available on the market for Raspberry pi, but i have a spare Raspberry PI 4B and i can connect this adapter and provide you SSH access to it if the issue is happening also on Raspberry PI 4B. If Raspberry PI 4B is not a good candidate i can provide access also to my rPI 5, but here we need to keep the running instance of zigbee2mqtt with USB - serial adapter and start a new instance on /dev/ttyAMA0 where i can plug the second zigbee controller.

ZigBee2MQTT is a nodejs based application and i tested both docker and host installation methods with same result. Both installation methods are provided here.

Regards!

[qrp73]

try to use /dev/ttyS0 instead of /dev/ttyAMA0. I remember, previously onboard serial port was worked as /dev/ttyS0 but when I added dtoverlay=disable-bt line into config.txt, it appears that serial port is reported as /dev/ttyS0 but cannot work with this name anymore and requires to use /dev/ttyAMA0 instead. So, may be you have similar issue.

[mariusmotea]

With dtoverlay=disable-bt i now have /dev/ttyS0 but i don't know the GPIO pind used, for sure are not 14 and 15. Also no change on ttyAMA0 functionality after adding dtoverlay=disable-bt

[pelwell]

disable-bt is not useful on a Pi 5 other than to disable Bluetooth - the UART that is freed cannot be mapped to the 40-pin header, and it's not like Pi 5 is short of UARTs that can be used for other purposes.

[pelwell]

There are two test kernels you can try:

• sudo rpi-update - this disables DMA to the UART
• sudo rpi-update pulls/6377 - this re-enables DMA in a way which is more aware of the UART's needs

[mariusmotea]

Hi,

I tested both and are working. Currently i'm on rpi-update pulls/6377 so this will be much better tested in the next days.

Thanks.

[pelwell]

Thanks for letting me know. One of those two changes will be released for all users.

[mgrouch]

Is it fixed in 6.6.62 and safe to upgrade to it?

Thanks

[pdbs]

Doesn't look like. Kernel

Linux cs5 6.6.51+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.51-1+rpt3 (2024-10-08) aarch64 GNU/Linux

is okay, but when you dist-upgrade, it's broken again. And rpi-update pulls/6377 didn't work for me. Gave me a 404.

[popcornmix]

And rpi-update pulls/6377 didn't work for me. Gave me a 404.

The github build artefacts are only kept for a limited time (maybe 1 month). Just use "sudo rpi-update" without pulls/6377. The latest build will include that PR (and possibly other useful ones).

Linux cs5 6.6.51+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.51-1+rpt3 (2024-10-08) aarch64 GNU/Linux

That is not the latest apt kernel for RPiOS bookworm. You should have: 6.6.62+rpt-rpi-2712 with a build date of 2024-11-25.

[6by9]

The github build artefacts are only kept for a limited time (maybe 1 month).

90 days, which is the maximum that Github allows https://github.com/raspberrypi/linux/blob/rpi-6.6.y/.github/workflows/kernel-build.yml#L108