Dynamic log level change doesn't work the same for all environments

[PhilipSchmid]

What happened?

The Tetragon doc has a Change log level dynamically section that explains how to dynamically change it using SIGRTMIN+x. However, SIGRTMIN isn't the same value for every environment:

The Linux kernel supports a range of 33 different real-time signals, numbered 32 to 64. However, the glibc POSIX threads implementation internally uses two (for NPTL) or three (for LinuxThreads) real-time signals (see pthreads(7)), and adjusts the value of SIGRTMIN suitably (to 34 or 35).

It depends on libc, and, e.g., Alpine uses musl: https://git.musl-libc.org/cgit/musl/commit/src/signal/sigrtmin.c?id=016a5dc1925a66c7d1ffc14b862b1342f399cfda

We would likely need to develop a proper tetra CLI command to set the debug level in a unified and environment-independent way.

I ran into this issue while deploying Tetragon on a Talos Linux environment. For reproduction, you could use https://github.com/isovalent/terraform-aws-talos/tree/main/example.

Relevant Tetragon part: https://github.com/isovalent/hubble-fgs/blame/4f51f6f7527b7486713b7f411340274bc7c3249d/cmd/tetragon/main.go#L521-L540

Tetragon Version

v1.1.0

Kernel Version

6.6.29-talos

Kubernetes Version

v1.29.3

Bugtool

No response

Relevant log output

No response

Anything else?

No response