Motivation

Sometimes we want to pass some values to the running process in a very lightweight way at runtime. Use cases include:

1. Enabling debugging logging on seemingly unaccessible process
2. Enable runtime profiling
3. Throttling of heavily loaded process

Background

1. memfd is a linux mechanism to communicate via shared memory in a safe way. Memfd are shared between processes as file descriptors. It's basically an anonymous temporary file.
2. We already have support for passing file descriptors for TCP sockets. While it's a bit hacky, it's tremendously useful for some applications

Solution

Lithos side:

1. Pass memfd to the process that contains some fixed-size-and-position equivalent of:

log-level: warn

2. Update log-level directly in the target memory

Application's task:

1. Map the respective memory position to in-code variable
2. Check it in appropriate places

The expected cost of reading a variable is a single atomic load instruction. So you can check the variable in every logging instruction. Even rust can afford it.

Applicability

Surely, this is not for full blown configs. We think that this feature needed for things that are:

1. Hard to reproduce on test cluster
2. Needs some feature enabled (like debug logging) that can't be enabled at all times
3. Has a setting that is easy to read to/from shared memrory
4. Or alternatively, requires immediate action which can't afford full service (or even config) reload: throttling, stop accepting connections, etc.

On the other hand, most things that traditionally where implemented via signals might be use this mechanism, because in most cases signal handlers do exactly the same thing: set some global flag and continue work until normal code flow checks the flag

Alternatives

1. Fetch the value of the variable by web API. The downsides are:
   • Cost of the request
   • Cost of the request library in the process that 99.9% of the time is useless
   • Can't enable fast enough: in synchronous process it will probably be too late when enabled, but in async process it may also be a problem if the process is heavily loaded. If you need to debug ENFILE error
2. Using signals such as SIGUSR1. The downsides are:
   • There are only few signals (there are real-time ones, but it's hard to find which are unused), so it may work as enable/disable, but not set the actual log level
   • It's hard to make sure signal is set up and will enable feature at right point in time (not after request or something)
   • EINTR handling still has issues in many languages and environments
3. Using just shared memory file:
   • Can't be sealed, so can potentially crash application
   • Not much simpler than memfd

/cc @anti-social, @popravich, @vharitonsky