mpaland-printf leads to broken output with GCC 9.3.1

[morigawa]

Description

Using gcc 9.3.1 mpaland-printf breaks the output on the nrf52840dk. (e.g. hello-world example). Changing to gcc 10.3.1 fixed the problem, but I assumed RIOT does support gcc 8+. Introduced was this issue in #21438, where the mpaland-printf package was added as dependency. Since this PR was merged, the output is broken. Adding that pkg manually to a makefile has the same effect at an older state.

Expected results

Without mpaland-printf:

2025-11-03 12:47:06,165 # main(): This is RIOT! (Version: 2025.10-devel-168-g2ed57-HEAD)
2025-11-03 12:47:06,165 # Hello World!
2025-11-03 12:47:06,170 # You are running RIOT on a(n) nrf52840dk board.
2025-11-03 12:47:06,173 # This board features a(n) nrf52 CPU.

Actual results

With mpaland-printf:

2025-11-03 12:47:49,442 # � � � � � � � � � � � � � �ou are running RIOT on a(n) nrf52840dk board.
2025-11-03 12:47:49,444 # This board features a(n) nrf52 CPU.

Versions

• Operating System: "Ubuntu" "22.04.5 LTS (Jammy Jellyfish)"
• arm-none-eabi-gcc: arm-none-eabi-gcc (GNU Arm Embedded Toolchain 9-2020-q2-update) 9.3.1 20200408 (release)
• clang: Ubuntu clang version 14.0.0-1ubuntu1.1

[mguetschow]

Maybe @maribu might have an idea what could cause this?

[maribu]

Interesting. I would suspect that the way we wrap the stdio functions to mpaland-printf on newlib not working with the used version of newlib.

Could you provide a docker container that contains the toolchain that builds the broken binary, so that I can reproduce the issue and test a potential fix with it?

If not, could you provide the ELF file that is affected (with debug symbols)?

[crasbe]

It was actually surprisingly difficult to get such an old version of GCC running. I did not find a package for it in the Launchpad, so I used the package from 20.04 Focal.

Dockerfile:

# Base image: Ubuntu 22.04 (Jammy Jellyfish)
FROM ubuntu:jammy

# Avoid interactive prompts during package installation
ENV DEBIAN_FRONTEND=noninteractive

# Update system and install required packages
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        ca-certificates \
        build-essential \
        libnewlib-arm-none-eabi \
        make \
        gcc-multilib \
        python3-serial \
        python3-psutil \
        wget \
        curl \
        unzip \
        git \
        openocd \
        gdb-multiarch \
        binutils-arm-none-eabi && \
    # Clean up to reduce image size
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

RUN wget http://launchpadlibrarian.net/465590960/libisl22_0.22.1-1_amd64.deb
RUN wget http://launchpadlibrarian.net/468792405/gcc-arm-none-eabi_9-2019-q4-0ubuntu1_amd64.deb

RUN dpkg -i libisl22_0.22.1-1_amd64.deb
RUN dpkg -i gcc-arm-none-eabi_9-2019-q4-0ubuntu1_amd64.deb

# Set a default working directory
WORKDIR /workspace

RUN git clone https://github.com/RIOT-OS/RIOT

# Default command when starting the container
CMD ["/bin/bash"]

buechse@skyleaf:~/RIOTstuff/docker-mpaland$ docker build -t docker-mpaland .
...

buechse@skyleaf:~/RIOTstuff/docker-mpaland$ docker run -it docker-mpaland
root@5e81085e03ce:/workspace# ls
RIOT
root@5e81085e03ce:/workspace# arm-none-eabi-gcc --version
arm-none-eabi-gcc (15:9-2019-q4-0ubuntu1) 9.2.1 20191025 (release) [ARM/arm-9-branch revision 277599]
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@5e81085e03ce:/workspace#

Edit: had to update it to include newlib.

[crasbe]

I had to update the Dockerfile to include newlib so it actually builds:

root@6ae9012a2c0d:/workspace/RIOT# BOARD=nrf52840dk make -C examples/basic/hello-world/ -j
make: Entering directory '/workspace/RIOT/examples/basic/hello-world'
Building application "hello-world" for "nrf52840dk" with CPU "nrf52".

"make" -C /workspace/RIOT/pkg/cmsis/
"make" -C /workspace/RIOT/pkg/mpaland-printf/
"make" -C /workspace/RIOT/build/pkg/mpaland-printf -f /workspace/RIOT/pkg/mpaland-printf/mpaland-printf.mk
...
"make" -C /workspace/RIOT/cpu/nrf5x_common/periph
   text    data     bss     dec     hex filename
   6940       0    2320    9260    242c /workspace/RIOT/examples/basic/hello-world/bin/nrf52840dk/hello-world.elf
make: Leaving directory '/workspace/RIOT/examples/basic/hello-world'

[morigawa]

Thank you @crasbe for providing the Dockerfile.

I tried that version (9.2.1) and it led to the same bug. The following could be used instead to get my exact version (9.3.1), but I guess it doesn't make a difference:

ARG ARM_URL=https://developer.arm.com/-/media/Files/downloads/gnu-rm/9-2020q2/gcc-arm-none-eabi-9-2020-q2-update-x86_64-linux.tar.bz2
ARG ARM_MD5=2b9eeccc33470f9d3cda26983b9d2dc6
ARG ARM_FOLDER=gcc-arm-none-eabi-9-2020-q2-update
RUN echo 'Installing arm-none-eabi toolchain from arm.com' >&2 && \
    mkdir -p /opt && \
    curl -L -o /opt/gcc-arm-none-eabi.tar.bz2 ${ARM_URL} && \
    echo "${ARM_MD5} /opt/gcc-arm-none-eabi.tar.bz2" | md5sum -c && \
    tar -C /opt -jxf /opt/gcc-arm-none-eabi.tar.bz2 && \
    rm -f /opt/gcc-arm-none-eabi.tar.bz2 && \
    echo 'Removing documentation' >&2 && \
    rm -rf /opt/gcc-arm-none-eabi-*/share/doc

ENV PATH ${PATH}:/opt/${ARM_FOLDER}/bin

[crasbe]

Depending on how much time @maribu wants to (or would have to) invest into this, we could also just exclude GCC < 10 for mpaland-printf. Not the nicest solution, but considering that Jammy already ships gcc-arm-none-eabi 10.3.1 and GCC 9 is EOL and GCC 9.3.1 is >5 years old, it might not be worth the effort.

[maribu]

@crasbe: Thx for providing the Dockerfile. I was able to reproduce the issue with

$ make BOARD=nrf52840dk -C examples/basic/hello-world BUILD_IN_DOCKER=1 DOCKER_IMAGE=
reproducer flash term

(I named the docker image reproducer build from the Dockerfile).

Disabling mpaland-printf with

diff --git a/cpu/cortexm_common/Makefile.features b/cpu/cortexm_common/Makefile.features
index 1e6056dcf3..6ed7df7cd1 100644
--- a/cpu/cortexm_common/Makefile.features
+++ b/cpu/cortexm_common/Makefile.features
@@ -1,6 +1,6 @@
 FEATURES_PROVIDED += arch_32bit
 FEATURES_PROVIDED += arch_arm
-FEATURES_PROVIDED += bug_newlib_broken_stdio
+#FEATURES_PROVIDED += bug_newlib_broken_stdio
 FEATURES_PROVIDED += cortexm_svc
 FEATURES_PROVIDED += cpp
 FEATURES_PROVIDED += cpu_check_address

fixes the issue. What is curious is that

diff --git a/examples/basic/hello-world/main.c b/examples/basic/hello-world/main.c
index c1fe9bae12..e298728ce3 100644
--- a/examples/basic/hello-world/main.c
+++ b/examples/basic/hello-world/main.c
@@ -17,10 +17,17 @@
  */
 
 #include <stdio.h>
+#include <string.h>
+#include "stdio_base.h"
 
 int main(void)
 {
+#if 1
+    const char *msg = "Hello World!\n";
+    stdio_write(msg, strlen(msg));
+#else
     puts("Hello World!");
+#endif
 
     printf("You are running RIOT on a(n) %s board.\n", RIOT_BOARD);
     printf("This board features a(n) %s CPU.\n", RIOT_CPU);

again results in broken stdio.

Stepping through the firmware generated with mpaland-printf does indicated that each and every call to stdio is properly wrapped to mpaland-printf and in GDB I see that each and every call to puts() and printf() results in a correctly formatted text being passed over to stdio_write() and later to uart_write().

In addition to that, using the reproducer image and building something for a different board results in:

2025-11-07 16:06:22,267 # main(): This is RIOT! (Version: 2026.01-devel-61-gb1256f)
2025-11-07 16:06:22,268 # Hello World!
2025-11-07 16:06:22,271 # You are running RIOT on a(n) nucleo-f767zi board.
2025-11-07 16:06:22,275 # This board features a(n) stm32 CPU.

So the preliminary result is that there likely is something wrong with uart_write() on the nRF52840, but it only gets triggered when using GCC 9.x and mpaland-printf.

[maribu]

I was so free to change the title, as the issue can be reproduced by calling stdio_write() directly without using (or even compiling in) mpaland-printf and as the issue is only affecting the nRF5x UART driver.

I recall that the UART driver on nRF5x did cause some strange behavior in the past. My guess is that code emitted by GCC 9.3.1 just a slightly different timing and that this triggers a bug in our periph_uart driver on nRF5x. So GCC 9.3.1 is not really "causing" the issue, but combined with either calling stdio_write() directly or using mpaland_print is just a reliable reproducer for a bug present in our periph_uart driver.

It would be good if someone(tm) finds the time to get to the root cause of the issue and fixes it for good.