RIOT Tools clean-up

Current situation

We have different tools that are used by the build system:

standard tools (sh, make)
tools we need installed (the various programmers)
tools we wrote ourselves and ship in source

(Compilers are somewhere between 1 and 2).

Installation of category 2 is handled with a bit of a mix that we should unify and address consistently – I think we should clean that up. This also affects category 3 when the script is not self-contained (eg. has Python dependencies).

Current approaches range between:

a) Just call the program and hope it is (or its dependencies are) installed (eg. bmp.inc.mk just calling dist/tools/bmp.py, hoping that some rare-ish Python modules are installed) b) Check whether the program is installed and err out if not (happens automatically for tools like $FLASHER) c) Provide for it to be installed inside our checkout through a pkg (bossa) d) Use an existing tool to specify what to load-on-demand (not in-tree AFAICT, but some PRs have "run `pipx --spec package program --args" in their "how to test")

This leads to bad user experience around "now I have to install XYZ, how do I even do that", but at the same time, other users would see their expectations violated if building RIOT suddenly touches something else on the system.

Boundaries for solutions

Anything we do to unify those should

reasonably ensure that even though we load code from the Internet, what gets installed is what a maintainer checked (well we don't audit all the way down, but at least it's what the maintainer thought would be safe to install on their system)

"reasonably" because while we may be able to say to install some package from git with a given checksum, that will often define its own dependencies in the form of versions against the package manager (eg. cargo install --git https://github.com/foo/bar --rev 0123456ff --locked will use that precise commit but still pull dependencies by version number from crates.io).
not touch outside of the RIOT dir, or only write to places that are known to be used for such purposes and overridable (eg. pipx will create a cache in ~/.cache or wherever XDG_CACHE points to, and that's fine).
This should work offline where it can; in particular, the riotbuild container should have everything installed, and not even check the network. (It's probably OK to check whether we're in a container and then do python $SCRIPT rather than pipx run $SCRIPT, probably).

Possible tools

pipx can do things for Python tools. It can do both script running (just have a # /// script / # dependencies = [ "foo", "bar" ] / # /// section after the shebang, and then the shebang can be #!/usr/bin/env python3 or #!/usr/bin/env -S pipx run), and running published tools as pipx run --spec 'aiocoap[oscore] aiocoap-client`
cargo itself can install files in other places with --root, but we might prefer doing something like cargo-quickinstall / cargo-binstall, which even has associated tools like cargo-run-bin that do something similar to pipx's run.
Packaging things up as a pkg can be an option; I'm not sure we'll need its power here.
@crasbe wrote something quite complete for a single case that could be refactored to work for many cargo installable tools (in https://github.com/RIOT-OS/RIOT/pull/21466#issuecomment-2880227758)

Previous discussion

Sending more Python scripts through pipx (essentially doing 3 with solution d, unless in the riotbuild, then just assume the dependencies are installed) was talked about in Friday chats and seems to have good support.

May 22 '25 08:05 chrysn

Something like ./install-all-the-required-crap.sh would be good to have.
Ideally, it takes a place to put it all. "." might be on NFS, or ramdisk, or.. I might also have many RIOT-OS trees, and I want the tools installed once. Ideally, it also takes "-n" which just looks for the tools, reports they are missing/bad-versioned, and exits, allowing me to fix this manually. (thank you)

May 22 '25 20:05 mcr

Something like ./install-all-the-required-crap.sh would be good to have.

Yes and no. I would recommend stepping back and first figure out what even is required at all? What does it mean to say "this is required for riot to work". It's tangential to "What boards/cpus does RIOT support?": A question that can not be answered at the moment because "RIOT support" is undefined. You can define it ad hoc as "if it is in-tree it is supported" and include board Foo in the list. But if I use Foo and realize: Feature bar is not implemented - I might feel lied to, by your answer.

So, if your install-all-the-things.sh installs the xtensa tool chain on my pc, I will be annoyed because I do not own xtensa cpus. No one should be pedantic on things like this, if it is reasonable to install something because it is small and 80% of users need it: "just eat it up" - a thing I can only say once we figured out what it means to be "required for riot to work".

May 23 '25 11:05 Teufelchen1

Something like ./install-all-the-required-crap.sh would be good to have.

Yes and no. I would recommend stepping back and first figure out what even is required at all? What does it mean to say "this is required for riot to work". It's tangential to "What boards/cpus does RIOT support?": A question that can not be answered at the moment because "RIOT support" is undefined. You can define it ad hoc as "if it is in-tree it is supported" and include board Foo in the list. But if I use Foo and realize: Feature bar is not implemented - I might feel lied to, by your answer.

Here, I mean that it compiles and can be installed. As discussed during the VMA, that doesn't mean it runs or blinks the expected LED.

So, if your install-all-the-things.sh installs the xtensa tool chain on my pc, I will be annoyed because I do not own xtensa cpus. No one should be pedantic on things like this, if it is reasonable to install something because it is small and 80% of users need it: "just eat it up" - a thing I can only say once we figured out what it means to be "required for riot to work".

Okay: ./install-all-the-required-crap.sh BOARD=foobar

May 23 '25 12:05 mcr

stepping back and first figure out what even is required at all?

That is a very complex question that may exceed what is practical with RIOT. There's the tools you need to build, those are pretty board specific, but then there's the programmer (often really board independent, liked do you wire up a JLink or a that ST probe or a BMP, and possibly a matter of taste), and tools that are more relevant on a developer-but-not-user level (hello codespell). Eventually, this would need modelling all the inputs to make (maybe ./install-for BOARD=particle-xenon PROGRAMMER=bmp -C examples/gcoap) to the point where you could just run that line ones too.

We do have a single list of "install-all-the-required-💩" that is the riotbuild dockerfile; as we enhance things around this issue, I'd hope that that script becomes a bit easier (maybe b/c we can gather all the Python requirements.txt), and indeed that's gigabytes.

Maybe there is room for something in-between; we'll probably learn more about that in the course of this issue, but I'd not focus too much on it.

May 23 '25 13:05 chrysn