tini
tini copied to clipboard
krallin
Reload on file change handling for tini-watched processes
Some container-based programs can do an inline reload when a config file (particularly those mounted on a Volume) changes, but need to be signalled. For a local docker, we can do docker kill and let the signal pass through tini, but on a scheduler, like Kubernetes, we either have to run commands in each container or delete and recreate the containers (as per the replication-group strategy).
With this patch, we can run something like prometheus or fluentd in containers, and be able to use kubernetes to push updates, with the relevant program auto-reloading, without the need for a side-bar watcher, making tini a little bit more like a "supervisor".
Hi @krallin, I apologise in advance for doing a bit of a drive-by PR here - I'm actually about to head off on holiday where I'm not likely to have internet access, but some of my colleagues are also watching this PR (I did it as part of my work), and may be able to help with any questions you have.
I've basically been wanting this for (in our environment), fluentd and prometheus in Kubernetes, although I'm sure we could do better with our kubectl automation to rotate the pods. Ideally, IMO, these kinds of services would automatically pick up their config changes, but that's not the world we live in, so allowing tini to do it is also helpful.
(Just a quick note that I've seen your PR and I'm spending a bit of time thinking through it, as this would be a new direction for Tini)
Hi @krallin, I'm back now to help. I appreciate that it's a new direction - but I think it's a valid one in a kubernetes world, and makes tini useful.
If we want to make it a bit nicer, it's plausible to do it by inotify (if available) rather than the polled stat() calls.
Hey @mb-m, thanks for your reply. I'm sorry I'm taking a while to respond to this — I have been pretty busy at work (and I will be out next week for vacation, so my responsiveness is going to get worse before it gets better 😄 ).
Can you tell me a little bit about how you're injecting these files into your containers using Kubernetes? I haven't worked much with Kubernetes so I'm not 100% sure how this is generally useful there.
Re: inotify: if this goes in, I'm fine with polling stat unless there's a fundamental reason not to: I'd rather keep Tini as simple as possible.
Cheers,
Hi @krallin, first off, thanks for getting back, and I hope you have a lovely holiday! In Kubernetes, but also in docker, you can end up with volume mounts which are in some way outside the container. In the case of kubernetes, these end up being a "configmap" which is a separate structure to the "deployment" structure of the pod of containers. These end up creating what is effectively a docker volume for a single file, meaning that you can trivially replace the file without stopping and restarting the container.
While the container itself can be stopped and started, this can be more disruptive (eg. in the case where the container is running your monitoring - the stop and the start will miss some data collection and may miss generation of an alert), so where possible, it's worth having the reload-in-place semantics.
This obviously isn't the most useful in every situation, and for many situations the correct behaviour is to do the stop and restart as you'd expect, but in terms of improving reliability there are some where keeping the process running if possible is worthwhile, and doing this automatically rather than manually even more so (especially if you're making the same update across multiple environments).
Ok; I've been giving this some thought and I'd like to pull this in. I think there's quite a bit to be done to make Tini a slightly more feature-complete init system (outside of "minimal mode"), with this, and a few other possible features like https://github.com/krallin/tini/pull/75 or auto-restart.
I'll take a clover look at the code you're proposing here! I think that if we want to support additional features like this, a bit of a re-architecting might be worth it — the current architecture is very much tied to its "reap zombies and that's it" functionality.
Just as a matter of feedback on how useful this feature would be.
Currently I'm evaluating the use of the NOMAD scheduler and I'm facing a challenge regarding the use of Spring Boot Java applications and dynamic generated secrets.
Since we'll be using dynamic secrets in our environment, we can't really stand for restarting our Java applications everytime a token changes. Since Spring Boot applications only accept refreshes to the Beans scope through an API call to the /refresh endpoint having a init process being able to kick off a custom command once a file is updated would be pretty awesome.
At this point, our only solution is to have a sidekick daemon (watcher in this case) to monitor the file and execute a curl http://localhost/refresh.
