obelisk
obelisk copied to clipboard
obsidiansystems
Backend does not reload on deployment if configuration changes
When only changing a config in common (I tested with config/common/route) a deploy won't reload the backend and it continues to use the old config. To fix this a login to the server and doing a manual systemctl restart backend is needed.
We just encountered this when doing a deploy to update our email configuration. The workaround is easy enough for now but it would be great if all deploys that changed anything on the server would restart (or if this is tricky, just all deploys would cause a restart).
To summarize discussions between myself, @ryantrinkle, @3noch, and @eskimor (see #469), the fundamental issue seems to be how to keep the NixOS system and config directory in synch.
Ideally, the backend would itself be able to monitor the config directory and be notified of updates, without any need for a restart whatsover. This would be precisely correct, and as a bonus minimize downtime for config changes, so would definitely be nice to support in the future.
As for our present situation, the nix store is unpermissioned, so we can't just store configs there, as it would be insecure in any shared cache. Encrypting the configs and then putting them in the nix store could work, but then there wouldn't be a good way for a user to see the current configuration on their server, and we don't want that.
Other possibilities include passing some representation of the config directory to the backend derivation. This would ensure synchronization at deploy time, and having hash-based symlinks to config directories as a recreation of the nix store just for the backend would be even more robust, but neither would be able to handle subsequent mutations. We also considered manually triggering a server restart based on whether rsync detects changes, but that breaks if the deployment fails between those steps, so only pushes the problem further.
All this to say, our current solutions all came up short in some respect, but we're definitely open to other ideas. And for now, it seems better to re-evaluate this when the landscape changes than try to implement a partial solution.
IMO using a hash-based file structure to mimic nix is the least-bad solution. You could give it very restrictive permissions and drop the write bit to make it clear that writing should not be done. ob deploy would always rewrite the configs so that after any given ob deploy the server's state is exactly what you'd expect. If someone mutates config on the server, it's probably because they're trying to debug anyway (since that config should immediately be copied back to the deployment setup). And mutating the config within a given hashed directory would actually work fine until it gets overwritten by ob deploy.
I.e. configs should not be mutable in general. Supporting a debugging use case seems permissible and this idea seems to have the best of both worlds in that regard. Even putting your config in /nix/store doesn't support debugging in this way.
Maybe the config dir should be a git repo?
On December 3, 2019 11:18:42 PM UTC, Elliot Cameron [email protected] wrote:
IMO using a hash-based file structure to mimic nix is the least-bad solution. You could give it very restrictive permissions and drop the write bit to make it clear that writing should not be done.
ob deploywould always rewrite the configs so that after any givenob deploythe server's state is exactly what you'd expect. If someone mutates config on the server, it's probably because they're trying to debug anyway (since that config should immediately be copied back to the deployment setup). And mutating the config within a given hashed directory would actually work fine until it gets overwritten byob deploy.-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/obsidiansystems/obelisk/issues/372#issuecomment-561401445
Isn't the config dir already effectively a git repo since it's part of the deployment repo? The deployment git repo is effectively nothing more than a src thunk + config.
