riotdocker Add tiny and small build images

Add tiny build images

This adds Alpine based images intended for use with

make BUILD_IN_DOCKER=1 DOCKER_IMAGE=docker.io/riot/tinybuild-<ARCH>

This adds the following images:

tinybuild-native64: C toolchain needed to build RIOT apps for native64
smallbuild-native64: C, C++, and rust toolchain needed to build RIOT apps for native64
tinybuild-arm: C toolchain needed to build RIOT apps for ARM7 & ARM Cortex M boards
smallbuild-arm: C, C++ and rust toolchain libc needed to build RIOT apps for ARM7 & ARM Cortex M boards
tinybuild-avr: C and C++ toolchain and AVR libc needed to build RIOT apps for AVR boards
tinybuild-msp430: C toolchain needed to build RIOT apps for MSP430 boards
smallbuild-msp430: C and C++ toolchain needed to build RIOT apps for MSP430 boards
tinybuild-risc-v: C toolchain needed to build RIOT apps for RISC-V boards
smallbuild-risc-v: C, C++, and rust toolchain needed to build RIOT apps for RISC-V boards

For comparison, this is the size:

REPOSITORY                            TAG         IMAGE ID      CREATED            SIZE
docker.io/riot/riotbuild              latest      138e78010e19  4 weeks ago        13.5 GB
localhost/maribu/smallbuild-arm       latest      1fb5420486b7  56 minutes ago     4.23 GB
localhost/maribu/smallbuild-base      latest      a56817ee290e  59 minutes ago     1.74 GB
localhost/maribu/smallbuild-msp430    latest      11e2ecc41466  36 minutes ago     452 MB
localhost/maribu/smallbuild-native64  latest      86a4a333ac44  48 minutes ago     2.39 GB
localhost/maribu/smallbuild-risc-v    latest      a4692906f483  39 minutes ago     3.28 GB
localhost/maribu/tinybuild-arm        latest      d3718b8a6b57  58 minutes ago     1.17 GB
localhost/maribu/tinybuild-avr        latest      41013f963ddc  37 minutes ago     405 MB
localhost/maribu/tinybuild-base       latest      bd9802e90d98  About an hour ago  62.7 MB
localhost/maribu/tinybuild-msp430     latest      9f2c69543069  36 minutes ago     342 MB
localhost/maribu/tinybuild-native64   latest      6d01bf6ebece  51 minutes ago     239 MB
localhost/maribu/tinybuild-risc-v     latest      97066835a715  37 minutes ago     1.06 GB

I tested to build examples/basic/default for one board with each docker image successfully.

Mar 26 '25 21:03 maribu

Nice!

Any thoughts already on how we could integrate this into our CI? Having separate docker images is certainly less straightforward. But even summing all of them up into a single container would end up with way less than what we currently have (maybe also just because some tools for, e.g., static tests are not included?). In any case I'd prefer not having both the old debian-based container and the new alpine-based container(s) around in the future to avoid confusion. What would be currently still missing to do the switch?

Or are those mostly meant to be used for local development instead of CI?

Mar 27 '25 14:03 mguetschow

A single container that would contain all listed toolchains is very much possible and ends up being about 2 GiB. I can also add the tools needed for static tests. I think we would end up with about 2.5 GiB then.

However, this will not be able to replace the current image:

There are no ESP toolchains packaged for Alpine. Using the prebuild magic Espressif toolchain with the glibc compat package installed might be OKish. Better would be to just package the Espressif Toolchain. I have spent 2 weeks on that and ended up with a toolchain that worked for the bootloader, but not for RIOT apps.
Alpine has no multilib support. An x86_64 container only supports native64 and not native32.
There are still bugs in native* on musl that need to be fixed. This is probably the easiest of the three.

So from the RIOT's PoV, those images will probably remain a local option for users to use with BUILD_IN_DOCKER=1.

That said, using them in a CI is for downstream projects highly interesting. The typical downstream project consists of one (or a few) boards, one (or a few) modules, and a RIOT git submodule. E.g. my business card project could very much make use of the tinybuild-arm container for a CI. (The project I intend to use this for is a bit more complex and larger, but has a somewhat similar architecture.)

I have some interest in maintaining small containers capable for powering the CI of a downstream project that only needs to build for a single arch.

Mar 27 '25 16:03 maribu

By the way... how do you test these locally? They aren't pushed to docker.io yet obviously, so the FROM statement fails.

I played around with it yesterday but didn't get very far. I'm not super experienced with Docker...

Nov 05 '25 13:11 crasbe

You can label them to e.g. like this:

docker buildx build . -t local/smallbuild-base:latest

and then use the DOCKER_REGISTRY argument, e.g.:

docker buildx build --build-arg DOCKER_REGISTRY=local . -t local/tinybuild-native64:latest

Nov 05 '25 13:11 maribu

I got it working with the aforementioned changes and my PR applied with the following modification:

diff --git a/makefiles/docker.inc.mk b/makefiles/docker.inc.mk
index fee33ac013..8ec546c8a0 100644
--- a/makefiles/docker.inc.mk
+++ b/makefiles/docker.inc.mk
@@ -8,7 +8,7 @@
 DOCKER_TESTED_IMAGE_REPO_DIGEST := 08fa7da2c702ac4db7cf57c23fc46c1971f3bffc4a6eff129793f853ec808736

 # "DOCKER_IMAGE_VARIANT" is a placeholder that is substituted by `$(DOCKER_IMAGE_VARIANT)`
-DOCKER_PULL_IDENTIFIER := docker.io/riot/DOCKER_IMAGE_VARIANT@sha256:$(DOCKER_TESTED_IMAGE_REPO_DIGEST)
+DOCKER_PULL_IDENTIFIER := local/DOCKER_IMAGE_VARIANT
 export DOCKER_BUILD_ROOT ?= /data/riotbuild
 DOCKER_RIOTBASE ?= $(DOCKER_BUILD_ROOT)/riotbase

buechse@skyleaf:~/RIOTstuff/riot-vanilla/RIOT$ BUILD_IN_DOCKER=1 BOARD=nrf52840dk make -C examples/basic/hello-world
make: Entering directory '/home/buechse/RIOTstuff/riot-vanilla/RIOT/examples/basic/hello-world'
Launching build container using image "local/tinybuild-arm".
Building application "hello-world" for "nrf52840dk" with CPU "nrf52".

"make" -C /data/riotbuild/riotbase/pkg/cmsis/
...
"make" -C /data/riotbuild/riotbase/sys/stdio_uart
   text    data     bss     dec     hex filename
   6700       0    2320    9020    233c /data/riotbuild/riotbase/examples/basic/hello-world/bin/nrf52840dk/hello-world.elf

Very nice :)

Nov 05 '25 14:11 crasbe

For reference these are the commands I used:

riotdocker$ docker buildx build ./tinybuild-apks -t local/tinybuild-apks:latest

riotdocker$ docker buildx build --build-arg DOCKER_REGISTRY="local" ./tinybuild-base/ -t local/tinybuild-base:latest

riotdocker$ docker buildx build --build-arg DOCKER_REGISTRY="local" ./tinybuild-arm/ -t local/tinybuild-arm:latest

Nov 05 '25 14:11 crasbe

https://github.com/RIOT-OS/riotdocker/actions/runs/19107811806

This is why the Build test is not executed, but I'm not sure why? Perhaps this needs a rebase, the only difference I could find was this line: https://github.com/RIOT-OS/riotdocker/blob/6662ca1c05c7944892c32cbb4fc244a8190fcf1f/.github/workflows/build.yml#L33 and the versions: https://github.com/RIOT-OS/riotdocker/blob/6662ca1c05c7944892c32cbb4fc244a8190fcf1f/.github/workflows/build.yml#L40-L41

Nov 05 '25 15:11 crasbe

I just realized that c2rust is now available in stable Alpine releases as well, so I could drop it from building in in tinybuild-apks ourselves.

However, I had to disable C++ on MSP430 a while ago, see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=119953

I'm retesting now to see if the bug has been fixed as side-effect of something. But I don't have high hopes. In the worst case, smallbuild-msp430 would need to be dropped for now.

Nov 08 '25 17:11 maribu

Nope, still no C++ for MSP430:

/home/maribu/Repos/software/aports/master/community/g++-cross-embedded/src/gcc-15.2.0/libstdc++-v3/src/c++17/fs_path.cc:2005:1error: insn does not satisfy its constraints:
 2005 | }
      | ^
(insn 39 87 88 3 (set (reg:SI 14 R14)
        (plus:SI (reg:SI 14 R14)
            (reg:SI 13 R13 [orig:48 _6 ] [48]))) "/home/maribu/Repos/software/aports/master/community/g++-cross-embedded/src/gcc-15.2.0/libstdc++-v3/src/c++17/fs_path.cc":2002:14 22 {addsi3}
     (nil))
during RTL pass: postreload
/home/maribu/Repos/software/aports/master/community/g++-cross-embedded/src/gcc-15.2.0/libstdc++-v3/src/c++17/fs_path.cc:2005:1internal compiler error: in extract_constrain_insn, at recog.cc:2783
0x7fe15c573293 libc_start_main_stage2
	src/env/__libc_start_main.c:95
Please submit a full bug report, with preprocessed source (by using -freport-bug).
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.
make[9]: *** [Makefile:587: cow-fs_path.lo] Error 1
make[9]: *** Waiting for unfinished jobs....
/home/maribu/Repos/software/aports/master/community/g++-cross-embedded/src/gcc-15.2.0/libstdc++-v3/src/c++17/fs_path.cc: In function 'std::size_t std::filesystem::__cxx11::hash_value(const path&)':
/home/maribu/Repos/software/aports/master/community/g++-cross-embedded/src/gcc-15.2.0/libstdc++-v3/src/c++17/fs_path.cc:2005:1error: insn does not satisfy its constraints:
 2005 | }
      | ^
(insn 38 86 87 3 (set (reg:SI 14 R14)
        (plus:SI (reg:SI 14 R14)
            (reg:SI 13 R13 [orig:48 _6 ] [48]))) "/home/maribu/Repos/software/aports/master/community/g++-cross-embedded/src/gcc-15.2.0/libstdc++-v3/src/c++17/fs_path.cc":2002:14 22 {addsi3}
     (nil))
during RTL pass: postreload
/home/maribu/Repos/software/aports/master/community/g++-cross-embedded/src/gcc-15.2.0/libstdc++-v3/src/c++17/fs_path.cc:2005:1internal compiler error: in extract_constrain_insn, at recog.cc:2783

Nov 08 '25 17:11 maribu

Or is there no C++ on MSP430 at all?

Older GCC releases still build g++ for MSP430. I think the issue might get solved in upstream at some point in time. Until then, riotboot still ships an old GCC version that still compiled with C++ enabled...

Nov 08 '25 18:11 maribu

That's ironic. No space left for tinybuild 😅

On which machine is that running? 🤔

Nov 08 '25 18:11 crasbe

On a Github runner.

It does make sense that those runners don't have unlimited storage. We mighty be able to split the tasks somehow, so that tinybuild related build tasks and functionality tests are done on independent workers?

Nov 09 '25 08:11 maribu

We mighty be able to split the tasks somehow, so that tinybuild related build tasks and functionality tests are done on independent workers?

Sounds like a good idea 👍

Nov 10 '25 10:11 crasbe

For reference these are the commands I used:

On Debian bookworm (oldstable) without docker-buildx, I had to apply the following patch:

diff --git a/tinybuild-arm/Dockerfile b/tinybuild-arm/Dockerfile
index b09d2d6..0a729f9 100644
--- a/tinybuild-arm/Dockerfile
+++ b/tinybuild-arm/Dockerfile
@@ -5,7 +5,7 @@ LABEL maintainer="Marian Buschsieweke <[email protected]>"
 
 RUN \
     --mount=type=cache,id=apk-cache,sharing=locked,target=/var/cache/apk \
-    --mount=type=bind,target=/tinybuild-apks,source=/output,from=tinybuild-apks \
+    --mount=type=bind,target=/tinybuild-apks,source=/output,from=local/tinybuild-apks \
     apk add \
         newlib-arm-none-eabi \
         picolibc-arm-none-eabi
diff --git a/tinybuild-base/Dockerfile b/tinybuild-base/Dockerfile
index e59f1ae..ef58927 100644
--- a/tinybuild-base/Dockerfile
+++ b/tinybuild-base/Dockerfile
@@ -4,7 +4,7 @@ LABEL maintainer="Marian Buschsieweke <[email protected]>"
 
 RUN \
     --mount=type=cache,id=apk-cache,sharing=locked,target=/var/cache/apk \
-    --mount=type=bind,target=/tinybuild-apks,source=/output,from=tinybuild-apks \
+    --mount=type=bind,target=tinybuild-apks,source=/output,from=local/tinybuild-apks \
     echo "@riotapks /tinybuild-apks/riotapks" >> /etc/apk/repositories && \
     cp /tinybuild-apks/*.rsa.pub /etc/apk/keys/ && \
     apk update && \

before building with

$ docker -v
Docker version 20.10.24+dfsg1, build 297e128
$ DOCKER_BUILDKIT=1 docker build ./tinybuild-apks -t local/tinybuild-apks:latest
$ DOCKER_BUILDKIT=1 docker build --build-arg DOCKER_REGISTRY="local" ./tinybuild-base/ -t local/tinybuild-base:latest
$ DOCKER_BUILDKIT=1 docker build --build-arg DOCKER_REGISTRY="local" ./tinybuild-arm/ -t local/tinybuild-arm:latest

Nov 26 '25 09:11 mguetschow

To run tests with BUILD_IN_DOCKER=1 on native64 (which are apparently by default also run inside the container, different to non-native targets), I additionally had to install py-pexpect for the testrunner:

diff --git a/tinybuild-native64/Dockerfile b/tinybuild-native64/Dockerfile
index 26447d0..fdc5b02 100644
--- a/tinybuild-native64/Dockerfile
+++ b/tinybuild-native64/Dockerfile
@@ -5,9 +5,10 @@ LABEL maintainer="Marian Buschsieweke <[email protected]>"
 
 RUN \
     --mount=type=cache,id=apk-cache,sharing=locked,target=/var/cache/apk \
-    --mount=type=bind,target=/tinybuild-apks,source=/output,from=tinybuild-apks \
+    --mount=type=bind,target=/tinybuild-apks,source=/output,from=local/tinybuild-apks \
     apk add \
         libucontext-dev@riotapks \
         gcc \
         musl-dev \
-        linux-headers
+        linux-headers \
+        py3-pexpect

Nov 26 '25 09:11 mguetschow

I've just went ahead and ran all tests for native64 on current RIOT master (EDIT: with https://github.com/RIOT-OS/RIOT/pull/21915 applied) within tinybuild-native64, i.e., BUILD_IN_DOCKER=1 DOCKER_IMAGE=local/tinybuild-native64:latest ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native64

See the full list of build and test failures below. The test failures are mostly due to missing python packages, namely scapy and riotctrl. tests/pkg/spiffs lead to a segmentation fault after spiffs_tests.tests_spiffs_open_close (tests/pkg/spiffs/main.c 121) exp 0 was -9. Maybe it's fine if the tinybuild containers cannot be used to run all tests.

Regarding build failures, we have:

All rust-related applications are expected to fail as tinybuild explicitly excludes the Rust toolchain.
Did we expect C++ applications to fail to build with make[1]: g++: No such file or directory?
all fuzzing tests fail because of missing afl-gcc
examples/networking/misc/lwm2m, tests/build_system/kconfig, tests/net/gcoap_dns, tests/net/gnrc_sixlowpan_frag*, tests/sys/congure_*, tests/sys/conn_can, tests/sys/progress_bar fail with /bin/sh: /data/riotbuild/riotbase/dist/tools/fixdep/fixdep: not found
tests/build_system/utils fail with make[1]: bash: No such file or directory
tests/pkg/flatbuffers, tests/pkg/relic, examples/lang_support/community/javascript, examples/lang_support/community/wasm, tests/pkg/cryptoauthlib, tests/pkg/cryptoauthlib_compare_sha256, tests/sys/psa_crypto_se*, examples/networking/misc/ccn-lite-relay fail because cmake is not installed
tests/pkg/lvgl, tests/pkg/lvgl_touch fail with /data/riotbuild/riotbase/tests/pkg/lvgl_touch/bin/native64/riotbuild/riotbuild.h:29:26: fatal error: SDL.h: No such file or directory
tests/cpu/native_backtrace fails with /data/riotbuild/riotbase/cpu/native/backtrace/backtrace.c:13:10: fatal error: execinfo.h: No such file or directory
tests/build_system/external_unittests, tests/unittests fail with

/usr/lib/gcc/x86_64-alpine-linux-musl/14.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find libasan_preinit.o: No such file or directory
/usr/lib/gcc/x86_64-alpine-linux-musl/14.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lasan: No such file or directory
collect2: error: ld returned 1 exit status

tests/build_system/external_board_native fails with /data/riotbuild/riotbase/core/lib/atomic_c11.c:77:10: error: conflicting types for built-in function '__atomic_load_8
tests/pkg/nanopb fails with /bin/sh: protoc: not found
examples/advanced/suit_update and tests/sys/suit_manifest fail with Choose encryption for key file /data/riotbuild/riotbase/tests/sys/suit_manifest/bin/native64/default.pem: /bin/sh: openssl: not found

So, in short:

Do we want to add a C++ compiler, afl-gcc, cmake and other dependencies (that might be needed by only very few tests) to the build container? What other options would we have? test-specific docker containers?
bash should probably not be called explicitly in the tests. That should be fixed in RIOT.
I would have to double-check where/how the fixdep tool is built. I would have expected the build system to figure out it is missing and build it on demand.
The __atomic_load_8 issue should be further investigated.
What about execinfo.h and libasan on Alpine?

Failures during compilation:

examples/advanced/suit_update
examples/lang_support/community/javascript
examples/lang_support/community/wasm
examples/lang_support/official/riot_and_cpp
examples/lang_support/official/rust-async
examples/lang_support/official/rust-hello-world
examples/networking/misc/ccn-lite-relay
examples/networking/misc/lwm2m
fuzzing/gcoap
fuzzing/gnrc_tcp
fuzzing/uri_parser
tests/build_system/cpp_exclude
tests/build_system/cpp_ext
tests/build_system/external_board_native
tests/build_system/external_unittests
tests/build_system/kconfig
tests/build_system/utils
tests/core/rmutex_cpp
tests/cpu/native_backtrace
tests/net/gcoap_dns
tests/net/gnrc_sixlowpan_frag_minfwd
tests/net/gnrc_sixlowpan_frag_sfr
tests/net/gnrc_sixlowpan_frag_sfr_congure
tests/pkg/cryptoauthlib
tests/pkg/cryptoauthlib_compare_sha256
tests/pkg/etl
tests/pkg/flatbuffers
tests/pkg/lvgl
tests/pkg/lvgl_touch
tests/pkg/nanopb
tests/pkg/relic
tests/pkg/tflite-micro
tests/pkg/utensor
tests/rust_libs
tests/rust_minimal
tests/sys/congure_abe
tests/sys/congure_quic
tests/sys/congure_reno
tests/sys/conn_can
tests/sys/cpp11_condition_variable
tests/sys/cpp11_mutex
tests/sys/cpp11_thread
tests/sys/cpp_ctors
tests/sys/progress_bar
tests/sys/psa_crypto_se
tests/sys/psa_crypto_se_cipher
tests/sys/psa_crypto_se_ecdsa
tests/sys/psa_crypto_se_mac
tests/sys/suit_manifest
tests/unittests

Failures during test:

tests/net/gnrc_netif_ieee802154
tests/pkg/libschc
tests/pkg/spiffs
tests/sys/congure_test
tests/turo

Nov 26 '25 12:11 mguetschow

bash should probably not be called explicitly in the tests. That should be fixed in RIOT.

Turns out bash is used in a lot of shell scripts as the shebang. Do we have a policy for that, and do we maybe want to relax it to use /bin/sh instead?

Nov 26 '25 13:11 mguetschow