rpi-eeprom icon indicating copy to clipboard operation
rpi-eeprom copied to clipboard

Floating UART0 RX line in bootloader on CM5/(Pi 5)

Open Frederik-Polen opened this issue 9 months ago • 14 comments

Describe the bug

UART0 Rx line seems to be floating. This is detected by receiving unwanted characters when Rx line is not connected to a transmitting device. This is a problem for running second staged bootloaders like U-boot, which is interrupted by receiving (random) characters

Steps to reproduce the behaviour

  1. Flash a working 2nd stage bootloader.
  2. Make sure no device is connected on uart0 pins
  3. Boot device
  4. Wait till 2nd stage bootloader should have handed control to the linux kernel.
  5. Connect terminal to uart
  6. Verify device is still in 2nd stage bootloader.

Device (s)

Other, Raspberry Pi 5

Bootloader configuration.

any

System

n/a

Bootloader logs

1.22 RPi: BOOTSYS release VERSION:f788aab6 DATE: 2025/02/12 TIME: 10:51:52 1.26 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1739357512 serial 8e57621c boardrev c04180 stc 1226381 1.36 AON_RESET: 00000003 PM_RSTS 00001000 1.40 POWER_OFF_ON_HALT: 1 WAIT_FOR_POWER_BUTTON 0 power-on-reset 1 1.50 RP1_BOOT chip ID: 0x20001927 1.58 part 00000000 reset_info 00000000 1.59 PMIC reset-event 00000000 rtc 00000001 alarm 00000000 enabled 0 1.66 uSD voltage 3.3V 1.72 SDRAM_CONFIG: 15 1.72 Initialising SDRAM rank 2 total-size: 32 Gbit 4267 (0x15 0x15) 1.78 DDR 4267 1 0 32 152 BL:1 3.47 OTP boardrev c04180 bootrom a a 3.48 Customer key hash 0000000000000000000000000000000000000000000000000000000000000000 3.55 VC-JTAG unlocked 3.78 RP1_BOOT chip ID: 0x20001927

3.15 RP1_BOOT chip ID: 0x20001927 3.15 RP1_BOOT: fw size 46888 4.53 PCI2 init 4.53 PCI2 reset 4.98 PCIe scan 00001de4:00000001 4.99 RP1_CHIP_INFO 20001927

4.01 RPi: BOOTLOADER release VERSION:f788aab6 DATE: 2025/02/12 TIME: 10:51:52 4.08 BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1739357512 serial 8e57621c boardrev c04180 stc 4408732 4.42 usb_pd_init status 2 4.42 Boot mode: SD (01) order 0 4.59 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 4.70 SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 4.79 EMMC 4.79 SD retry 1 oc 0 4.03 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276 4.15 OCR c0ff8080 [0] CID: 00150100424a54443452031c424a491b 4.31 SD HOST: 200000000 CTL0: 0x00800f04 BUS: 25000000 Hz actual: 25000000 HZ div: 8 (4) status: 0x1fff0000 delay: 4 4.42 SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2 4.51 MBR: 0x00002000, 131072 type: 0x0c 4.54 MBR: 0x00022000, 262144 type: 0x83 4.57 MBR: 0x00000000, 0 type: 0x00 4.61 MBR: 0x00000000, 0 type: 0x00 4.65 Trying partition: 0 4.68 type: 32 lba: 8192 'mkfs.fat' ' rpi-cm5 ' clusters 129022 (1) 4.74 rsc 32 fat-sectors 1009 root dir cluster 2 sectors 0 entries 0 4.80 FAT32 clusters 129022 4.85 [sdcard] autoboot.txt not found 4.86 Select partition rsts 0 C(boot_partition) 0 EEPROM config 0 result 1 4.93 Trying partition: 1 4.96 type: 32 lba: 8192 'mkfs.fat' ' rpi-cm5 ' clusters 129022 (1) 4.02 rsc 32 fat-sectors 1009 root dir cluster 2 sectors 0 entries 0 4.08 FAT32 clusters 129022 4.16 Read config.txt bytes 2570 hnd 0x6b 4.18 [sdcard] pieeprom.upd not found 4.20 usb_max_current_enable default 0 max-current 1500 4.32 Read bcm2712-rpi-cm5-cm5io.dtb bytes 77995 hnd 0xaa05 4.35 dt-match: compatible: raspberrypi,5-compute-model match: brcm,bcm2712 4.41 dt-match: compatible: brcm,bcm2712 match: brcm,bcm2712 4.47 Model is CM5. Selecting USB high current limit 4.58 Read /config.txt bytes 2570 hnd 0x6b 4.64 Read /config.txt bytes 2570 hnd 0x6b 4.69 MESS:00:00:04.869821:0: Initial voltage 800000 temp 36730 5.70 MESS:00:00:05.070214:0: avs_2712: AVS pred 8529 852900 temp 38379 5.73 MESS:00:00:05.073821:0: vpred 852 mV +0 5.82 MESS:00:00:05.082487:0: FB framebuffer_swap 1 5.02 MESS:00:00:05.102460:0: Select resolution HDMI0/2 hotplug 0 max_mode 2 5.06 MESS:00:00:05.106511:0: Select resolution HDMI1/2 hotplug 1 max_mode 2 5.25 MESS:00:00:05.125778:0: HDMI1 edid block 0 offset 0 5.28 MESS:00:00:05.128159:0: 00ffffffffffff0010acdad04c525744 5.33 MESS:00:00:05.133832:0: 011d010380351e78ea0565a756529c27 5.39 MESS:00:00:05.139504:0: 0f5054a54b00714f8180a9c0d1c00101 5.45 MESS:00:00:05.145177:0: 010101010101023a801871382d40582c 5.50 MESS:00:00:05.150850:0: 45000f282100001e000000ff00483458 5.56 MESS:00:00:05.156523:0: 425954320a2020202020000000fc0044 5.62 MESS:00:00:05.162196:0: 454c4c205032343139480a20000000fd 5.67 MESS:00:00:05.167869:0: 00384c1e5311000a20202020202001ca 5.85 MESS:00:00:05.185915:0: HDMI1 edid block 1 offset 128 5.88 MESS:00:00:05.188473:0: 020317b14c9005040302071601141f12 5.94 MESS:00:00:05.194146:0: 1365030c001000023a801871382d4058 5.99 MESS:00:00:05.199819:0: 2c45000f282100001e011d8018711c16 5.05 MESS:00:00:05.205491:0: 20582c25000f282100009e011d007251 5.11 MESS:00:00:05.211164:0: d01e206e2855000f282100001e8c0ad0 5.16 MESS:00:00:05.216837:0: 8a20e02d10103e96000f282100001800 5.22 MESS:00:00:05.222510:0: 00000000000000000000000000000000 5.28 MESS:00:00:05.228183:0: 0000000000000000000000000000003d 5.33 MESS:00:00:05.233874:0: HDMI1: best-mode 2 (limit 2) 1920x1080 60 Hz CEA modes be005d80000000000000000000000000 extensions 1 5.45 MESS:00:00:05.245495:0: FB0 disp 1 max-fb 1 1920x1080 stride 3840 base 0x3f800000 5.63 Loading 'bcm2712-rpi-cm5-cm5io.dtb' to 0x00000000 offset 0x100 5.73 Read bcm2712-rpi-cm5-cm5io.dtb bytes 77995 hnd 0xaa05 5.97 [sdcard] /overlays/overlay_map.dtb not found 5.99 /overlays/overlay_map.dtb 5.05 Read /overlays/bcm2712d0.dtbo bytes 1491 hnd 0xaa9e 5.30 MESS:00:00:05.330271:0: Loaded overlay 'bcm2712d0' 5.31 Read /config.txt bytes 2570 hnd 0x6b 5.33 [sdcard] /overlays/vc4-kms-v3d.dtbo not found 5.37 /overlays/vc4-kms-v3d.dtbo 5.19 Read /cmdline.txt bytes 104 hnd 0x6a 5.74 BMD "armstub8-2712.bin" not found 5.74 fs_open: 'armstub8-2712.bin' 5.78 Loading 'kernel_2712.img' to 0x00000000 offset 0x200000 5.14 Read kernel_2712.img bytes 628552 hnd 0xaaa1 5.16 PCI1 reset 5.28 PCI2 reset 5.38 set_reboot_order 0 5.38 set_reboot_arg1 0 5.39 USB-OTG disconnect 5.85 MESS:00:00:05.885662:0: Starting OS 5885 ms 5.91 MESS:00:00:05.891183:0: 00000040: -> 00000480 5.93 MESS:00:00:05.893031:0: 00000030: -> 00100080 5.97 MESS:00:00:05.897744:0: 00000034: -> 00100080 5.02 MESS:00:00:05.902457:0: 00000038: -> 00100080 5.07 MESS:00:00:05.907170:0: 0000003c: -> 00100080

NOTICE: BL31: v2.6(release):v2.6-240-gfc45bc492 NOTICE: BL31: Built : 12:55:13, Dec 4 2024

U-Boot 2024.07 (Jul 01 2024 - 18:07:18 +0000)

DRAM: 1016 MiB (effective 4 GiB) RPI: Board rev 0x18 outside known range RPI Unknown model (0xc04180) Core: 22 devices, 11 uclasses, devicetree: board MMC: mmc@fff000: 0, mmc@1100000: 1 Loading Environment from FAT... Unable to read "uboot.env" from mmc0:1... In: serial,usbkbd Out: serial,vidconsole Err: serial,vidconsole Net: No ethernet found.

starting USB... No USB controllers found Hit any key to stop autoboot: 0 U-Boot>

USB boot

n/a

NVMe boot

n/a

Network (TFTP boot)

n/a

Frederik-Polen avatar Feb 27 '25 16:02 Frederik-Polen

So there's no misunderstanding, what do you mean by UART0 and its pins?

pelwell avatar Feb 27 '25 16:02 pelwell

Hi pelwell, By UART0, I mean the uart, used by the First Stage Bootloader to output the bootloader logs. The 3 pins (RX, TX and GND) are found on the pads of the unmounted connecter of the RPI CM5 board.

Frederik-Polen avatar Feb 27 '25 16:02 Frederik-Polen

The bootloader doesn't use RX. U-boot etc should configure RX as desired.

timg236 avatar Feb 27 '25 18:02 timg236

If BOOT_UART=1 is enabled then the bootloader behaviour can be improved to leave the pulls in expected state. Although, the comment still applies that subsequent boot stages should not rely on early boot using or configuring UART pads at all i.e. BOOT_UART=0 should probably not change chip default FN-select or pulls as on Pi4.

timg236 avatar Feb 27 '25 19:02 timg236

I do understand that Rx is not used by the bootloader but it is still using it as part of the UART. So from this point of view, one could say that the uart its other pins should be in a consistent state, Maybe in the future, the bootloader might be interruptable like u-boot too. I was wondering if a dtbo could help in configuring a pullup on the Rx pin. I also checked my boot.conf, but it was already using BOOT_UART=1.

PS: Thanks for the quick replies!

Frederik-Polen avatar Feb 27 '25 20:02 Frederik-Polen

A bootloader with RX receive is a massive security vulnerability and has been exploited in u-boot before so that's ruled out. However, you can press space from a USB keyboard to invoke the boot-menu for alternate boot-modes.

timg236 avatar Feb 27 '25 20:02 timg236

Hello timg236,

I've tested with the latest pieeprom (2025-03-03), but the problem still persists. I think we will need to modify u-boot to add pull-up on the rx gpio pin of the uart of the bcm2712. But we do not have the datasheet of the bcm2712, so we do not know which registers to set to have this pull-up activated. Is this something you could help us with?

Frederik-Polen avatar Mar 05 '25 10:03 Frederik-Polen

The GPIO pins on the Pi 5 are controlled by the RP1 chip, which has a partial datasheet here and which additionally links to https://github.com/raspberrypi/linux/blob/rpi-6.12.y/drivers/pinctrl/pinctrl-rp1.c You might also find the sourcecode of https://github.com/raspberrypi/utils/blob/master/pinctrl/README.md to be useful.

lurch avatar Mar 05 '25 10:03 lurch

This is a BCM2712 pin, it should be covered in the kernel pinctrl driver. I don't believe there is a datasheet.

timg236 avatar Mar 05 '25 10:03 timg236

Ooops, sorry for the red herring! 🤦

lurch avatar Mar 05 '25 10:03 lurch

But it's supported by pinctrl - https://github.com/raspberrypi/utils/blob/master/pinctrl/gpiochip_bcm2712.c.

pelwell avatar Mar 05 '25 10:03 pelwell

Can you tell me which gpio of the bcm2712 soc the uart rx pin is on? Maybe also the register I need to write to and the value I need to set to enable the pull-up? Thanks.

Frederik-Polen avatar Mar 05 '25 11:03 Frederik-Polen

$ pinctrl | grep XD
14: a4    pn | hi // GPIO14 = TXD0
15: a4    pu | hi // GPIO15 = RXD0
126: a4    pn | hi // BT_TXD/GPIO26 = UART_TXD_0
127: a4    pu | hi // BT_RXD/GPIO27 = UART_RXD_0
213: a2    pu | hi // UART_TX_FS/AON_GPIO13 = VC_TXD0
214: a3    pu | hi // UART_RX_FS/AON_GPIO14 = VC_RXD0

So GPIO 14 in the aon_gpio controller.

$ pinctrl -v | head
GPIO chips:
  1f000d0000: rp1 (54 gpios)
  107d508500: brcmstb (36 gpios)
  107d517c00: brcmstb (38 gpios)
  107d504100: bcm2712d0 (0 gpios)
  107d510700: bcm2712d0_aon (0 gpios)
 0: ip    pu | hi // ID_SDA/GPIO0 = input
 1: ip    pu | hi // ID_SCL/GPIO1 = input
 2: no    pu | -- // GPIO2 = none
 3: no    pu | -- // GPIO3 = none

The pairing between pinctrl and GPIO blocks on 2712 is not so obvious, but from the number of GPIOs or the proximity of addresses you would probably work out that it is the GPIO controller at 0x107d517c00 and the pinctrl/pads at 0x107d510700 that you are interested in. Note from https://github.com/raspberrypi/utils/blob/master/pinctrl/gpiochip_bcm2712.c#L487 that the pull is a property of the pad, not the GPIO.

pelwell avatar Mar 05 '25 11:03 pelwell

After doing some detailed debugging with a scope and a hardware guy, we saw that the RX line was pulled low for a moment in the first stage bootloader. We could confirm this RX pin being pulled low ( or removing the pull-up) caused the unexpected character in the rx buffer which stopped u-boot from loading the kernel. We added some buffer flushing code in u-boot at initialisation and this confirmed our hypothesis.

The following code was modified in main.c of u-boot:

/* We come here after U-Boot is initialised and ready to process commands */
void main_loop(void)
{
	const char *s;

	bootstage_mark_name(BOOTSTAGE_ID_MAIN_LOOP, "main_loop");

	if (IS_ENABLED(CONFIG_VERSION_VARIABLE))
		env_set("ver", version_string);  /* set version variable */

	cli_init();

/***
*** START WORKAROUND
***/
	while (tstc()) {	/* we got an unintentional key press	*/
		getchar();	/* flush input	*/
		puts("Flushing unintentional character from input buffer\n.");
	}
/***
*** END WORKAROUND
***/

	if (IS_ENABLED(CONFIG_USE_PREBOOT))
		run_preboot_environment_command();
 

Frederik-Polen avatar Mar 06 '25 13:03 Frederik-Polen